Hitchhiker's Guide to Visual Studio and SQL Server, 7th Edition: Best Practice Architectures and Examples

Next Page

Book Cover
Hitchhiker's Guide to Visual Studio and SQL Server, 7th Edition: Best Practice Architectures and Examples
By R. Vaughn William, Peter Blackburn
...............................................
Publisher: Addison Wesley Professional
Pub Date: November 02, 2006
Print ISBN-10: 0-321-24362-5
Print ISBN-13: 978-0-321-24362-1
Pages: 1128

Table of Contents | Index
Overview

Since 1994 when he wrote his first "Hitchhiker's Guide", William Vaughn has been providing developers all over the world the intimate details of how SQL Server can be accessed and managed from RAD languages like Visual Basic and Visual Basic .NET. With the 7th Edition, Bill has completely rewritten this encyclopedic work from cover to covergiving readers his insightful views on how applications should be built to maximize both developer and code performance. Visual Studio and the languages it hosts have never been as sophisticated as they are todaythe same can be said for SQL Server. This makes it even more important for developers to understand how to best leverage their features without being held back by their complexity. That's what this book is all aboutmaking it easier for developers regardless of their know-how.

The 7th edition is unique in that it's designed to provide not only up-to-date tutorials on the latest development tools provided by Visual Studio and SQL Server, but also a solid platform of architectural advice and rich examples for developers trying to choose between the myriad of platform options. Beginners and experts alike will find comprehensive step-by-step instructions that can make the reader's introduction to the latest versions of Visual Studio and SQL Server far easier.

Key topic coverage includes:

• Data access architectures and how to choose the best strategy for Windows Forms, ASP.NET, XML Web Services, and SQL Server CLR executables. Where

do these make sense and how much will they cost to build and maintain?

• SQL Server and relational database fundamentals and inner-machinery.

How does SQL Server work and why is it important that developers know?

• Making the development experience more productive through judicious use of the Visual Studio toolset, and how to know when the wizards can help.

• Using the latest ADO.NET data provider efficiently and safely.

• How to protect the security of your databaseand your jobby avoiding common mistakes.

• How to build secure, efficient, scalable applications in less time with fewer resourceshow to create faster code faster.

• How to leverage the potential of SQL Server CLR executables and knowing

when these features make sense.

• How to work with your DBA to maintain database integrity and security.

• Working with the new Visual Studio report controls to expose your

organization's data safely and easily with or without leveraging existing

SQL Server Reporting Services technology.

William R. Vaughn is the President of Beta V Corporation and a Microsoft MVP. In 2000 he retired from Microsoft after 14 years to focus on mentoring, speaking, and writing. His specialty is data access application design especially when connecting to SQL Serverover the years he's written 12 books on the subject. He's a popular speaker at technical conferences all over the world where his wit and no-holds-barred technical insights win him rave reviews. William is also a member of the prestigious INETA Speaker's Bureau. His works include articles for SQL Server Magazine and a bi-weekly editorial for Processor Magazine as well as books published by Microsoft Press and Apress.

The book includes a DVD that contains a wealth of examples as well as several sample databases used to illustrate points discussed in the book. Authenticated readers will also have unrestricted access to the book's supporting web site, www.hitchhikerguides.net, where additional examples, online forums, and other supplementary materials are available.

www.awprofessional.com/msserverseries

www.hitchhikerguides.net

www.betav.com/blogs/billva

www.betav.com

Next Page


Table of Contents

Previous Page
Next Page

Book Cover
Hitchhiker's Guide to Visual Studio and SQL Server, 7th Edition: Best Practice Architectures and Examples
By R. Vaughn William, Peter Blackburn
...............................................
Publisher: Addison Wesley Professional
Pub Date: November 02, 2006
Print ISBN-10: 0-321-24362-5
Print ISBN-13: 978-0-321-24362-1
Pages: 1128

Table of Contents | Index

Copyright
Microsoft Windows Server System Series
Foreword
Acknowledgments
About the Authors
Introduction
 | How Did I Get to This Point?
 | The 7th Edition's Scope
 | Keeping Current
 | Is This Book Written For You?
 | Getting Help and Support
Chapter 1. Exploring Application Architectures
 | Introduction
 | Choosing the "Right" Architecture
 | Understanding Your Toolset
 | Recognizing Application Design Constraints
 | Choosing the Right Data Access Interface
 | Choosing the Right Database Management System
 | Understanding Basic Data Access Architectures
 | Summary
Chapter 2. How Does SQL Server Work?
 | Introduction
 | Servers and Versions
 | Installing SQL Server
 | Running SQL Server Services
 | Touring the SQL Server System Databases
 | Understanding the SQL Server Security System
 | Managing SQL Server Connections
 | Using the SQL Query Tools
 | Creating SELECT Queries
 | Creating and Querying Views
 | Implementing Business Rules
 | User-Defined (Aliased) Types
 | Implementing Constraints
 | Managing Databases and Queries with Batches and Scripts
 | The Query Optimizer and the Query Plan
 | Understanding the Buffer Cache
 | Executing Action Commands
 | Introducing Stored Procedures
 | Introducing Triggers
 | Using Transactions to Protect Data Integrity
 | Administrative Functions
 | Working with the Transaction Log
 | Summary
Chapter 3. Relational Databases 101
 | Introduction
 | Getting Started with Solid Database Design
 | Understanding Relational Database Normalization
 | Creating Tables, Rows, and Columns
 | Summary
Chapter 4. Getting Started with Visual Studio
 | Introduction
 | How I Got Here
 | Installing the "Right" Version of Visual Studio
 | Launching Visual Studio
 | Customizing Visual Studio 2005
 | Configuring Server (or Database) Explorers
 | Creating and Managing Database Connections
 | Managing Database Objects with the Server Explorer
 | Managing Queries with the Query Designer
 | Using the Server Explorer to View Synonyms
 | Using the Server Explorer to Manage Types
 | Using the Server Explorer to Manage Assemblies
 | Using the Server Explorer to Manage Servers
 | Summary
Chapter 5. Managing Executables with the Server Explorer
 | Introduction
 | Creating and Editing Stored Procedures
 | Creating New Stored Procedures
 | Executing T-SQL with Run Selection
 | Debugging Stored Procedures
 | Debugging Stored Procedures on Remote Instances
 | Debugging Stored Procedures from Code
 | Using the Server Explorer to Manage Functions
 | Summary
Chapter 6. Building Data Sources, DataSets, and TableAdapters
 | Why Create YADAI?
 | Is Strongly Typed Data Important?
 | What Is a Data Source?
 | What Is a TableAdapter?
 | What's Missing in the TableAdapter?
 | Creating Database-Based Data Sources
 | Configuring a TableAdapter
 | Binding to the TableAdapter Using Drag and Drop
 | Using a TableAdapter DirectlyWithout Drag-and-Drop?
 | Managing DataTable Classes in the Data Source Designer
 | Moving Data Sources Between Applications
 | Creating Web Service Data Sources
 | Summary
Chapter 7. Managing Data Tools and Data Binding
 | Introduction
 | Touring the Visual Studio Toolbox
 | Using the DataSet Toolbox Element
 | Introduction to Data Binding
 | Using the BindingSource Class
 | Using the BindingNavigator Control
 | Using the DataGridView Control
 | Using the ProgressBar Control
 | Summary
Chapter 8. Getting Started with ADO.NET
 | Introduction
 | Approaching Data Access Challenges
 | ADO.NET from 50,000 Feet
 | Using the Visual Studio Object Browser to Explore ADO.NET
 | Instantiating ADO.NET Objects
 | Exploring the System.Data.SqlClient Namespace
 | Exploring the SqlClient Namespace
 | Exploring the System.Data Namespace
 | Summary
Chapter 9. Getting Connected
 | Introduction
 | Connectivity StrategiesThat Include Security
 | Configuring the Server and Firewall
 | Connection Strategies
 | Establishing a Connection
 | Writing Code to Create Connections
 | Understanding and Managing the Connection Pool
 | Building a ConnectionString for Other Providers
 | Getting Visual Studio to Build Your ConnectionString
 | Opening and Closing Connections (Semi-) Automatically
 | Understanding the Connection Properties
 | Using the Connection Methods
 | Handling Connection Events
 | Managing Connection Exceptions
 | Summary
Chapter 10. Managing SqlCommand Objects
 | Introduction
 | Creating SqlCommand Objects
 | Integrating Ad Hoc Queries into Your Application
 | Coding Parameter Queries
 | Validating the Value
 | Executing Stored Procedures
 | Summary
Chapter 11. Executing SqlCommand Objects
 | Introduction
 | Executing SqlCommand Objects Synchronously
 | Understanding the SqlDataReader
 | Populating Data Structures
 | Fetching Rows Asynchronously
 | Executing Commands Asynchronously
 | Summary
Chapter 12. Managing Updates
 | Introduction
 | Using Wizards to Generate Update Action Commands
 | Update Alternatives: Using the TableAdapter Configuration Wizard
 | Complex Updates with Server-Side Logic
 | Managing Concurrency by Design or Collision
 | Summary
Chapter 13. Managing SQL Server CLR Executables
 | Introduction
 | Evolving CLR Executables Technology
 | Where Do CLR Executables Make Sense?
 | Designing a CLR Executable
 | Building Your First CLR Executable
 | Building and Deploying a CLR Executable
 | Setting Up a Test Environment
 | Working with ADO.NET from Within CLR Executables
 | Coding CLR Stored Procedures
 | Implementing Basic User-Defined Type (UDT) CLR Executables
 | Implementing Advanced CLR User Defined Types
 | Accessing CLR UDTs Elsewhere
 | Implementing CLR Aggregate Executables
 | Implementing CLR Triggers
 | Advanced Debugging
 | UDT Safety and IP Security
 | Summary
Chapter 14. Creating and Managing Reports
 | Introduction
 | Understanding Reporting Services and RDL
 | Visual Studio 2005 Reporting
 | What Are the Visual Studio 2005 Report Tools?
 | Building Your First Report
 | Exploring the ReportViewer Class in Depth
 | Managing Server Reports
 | Managing Parameters
 | Advanced Reporting Techniques
 | Implementing the Matrix Report
 | Summary
Chapter 15. Summary and Wintry: Where We Are Now
Appendix I. Installing the Examples and Test Databases
 | Installing the Examples
 | Installing the Example Databases
 | Chapter-Specific Configuration Issues
 | Summary
Appendix II. Reinstalling the DACW and Other Missing Functionality in Visual Studio
Appendix III. Monitoring SQL Server
 | Monitoring SQL Server with the SQL Profiler
 | Monitoring SQL Server and ADO.NET with Performance Counters
 | Summary
Appendix IV. Creating and Managing Server-Side Cursors
 | Why Are Server-side Cursors Important?
 | How Does ADO.NET Implement Cursors?
 | How Are Server-Side Cursors Managed?
 | How Can ADO.NET Create a Server-Side Cursor?
 | Fetching Data from the Cursor
 | Updating with a Server-Side Cursor
 | Summary
Index

Previous Page


Copyright

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and the publisher was aware of a trademark claim, the designations have been printed with initial capital letters or in all capitals.

The authors and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained herein.

The publisher offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales, which may include electronic versions and/or custom covers and content particular to your business, training goals, marketing focus, and branding interests. For more information, please contact:

U.S. Corporate and Government Sales
(800) 382-3419
corpsales@pearsontechgroup.com

For sales outside the United States please contact:

International Sales
international@pearsoned.com

Visit us on the Web: www.awprofessional.com

Library of Congress Cataloging-in-Publication Data:

Vaughn, William R.
Hitchhiker's guide to Visual studio and SQL server : best practice architectures and examples / William R.
Vaughn, Peter Blackburn.
p. cm.
ISBN 0-321-24362-5 (pbk. : alk. paper) 1. Application softwareDevelopment. 2. Microsoft Visual studio. 3.
SQL server. I. Blackburn, Peter, 1967- II. Title.

QA76.76.A65V39 2006
005.3dc22
2006030355

Copyright © 2007 Pearson Education, Inc.

All rights reserved. Printed in the United States of America. This publication is protected by copyright, and permission must be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or likewise. For information regarding permissions, write to:

Pearson Education, Inc.
Rights and Contracts Department
75 Arlington Street, Suite 300
Boston, MA 02116
Fax: (617) 848-7047

Text printed in the United States on recycled paper at R.R. Donnelley and Sons in Crawfordsville, Indiana. First printing, November 2006

Dedication

To Fran and Jack Vaughn and to all that have fallen in service of their country.


Microsoft Windows Server System Series

Books in the Microsoft Windows Server System Series are written and reviewed by the world's leading technical authorities on Microsoft Windows technologies, including principal members of Microsoft's Windows and Server Development Teams. The goal of the series is to provide reliable information that enables administrators, developers, and IT professionals to architect, build, deploy, and manage solutions using the Microsoft Windows Server System. The contents and code of each book are tested against, and comply with, commercially available code. This series should be an invaluable resource for any IT professional or student working in today's Windows environment.

Titles in the Series

Paul Bertucci, Microsoft SQL Server High Availability, 0-672-32625-6 (Sams)

Peter Blackburn and William R. Vaughn, Hitchhiker's Guide to SQL Server 2000 Reporting Services, 0-321-26828-8 (Addison-Wesley)

William Boswell, Learning Exchange Server 2003, 0-321-22874-X (Addison-Wesley)

Roberta Bragg, Windows Server 2003 Security, 0-321-30501-9 (Addison-Wesley)

Eric L. Brown, SQL Server 2005 Distilled, 0-321-34979-2 (Addison-Wesley)

Bill English, Olga Londer, Shawn Shell, Todd Bleeker, and Stephen Cawood, Microsoft Content Management Server 2002: A Complete Guide, 0-321-19444-6 (Addison-Wesley)

John C. Hancock and Roger Toren, Practical Business Intelligence with SQL Server 2005, 0-321-35698-5 (Addison-Wesley)

Don Jones, Managing Windows® with VBScript and WMI, 0-321-21334-3 (Addison-Wesley)

Sakari Kouti and Mika Seitsonen, Inside Active Directory, Second Edition: A System Administrator's Guide, 0-321-22848-0 (Addison-Wesley)

Jason Nadrowski and Stacy Draper, SharePoint 2003 Advanced Concepts, 0-321-33661-5 (Addison-Wesley)

Shyam Pather, Microsoft SQL Server 2000 Notification Services, 0-672-32664-7 (Sams)

Michael Raheem, Dima Sonkin, Thierry D'Hers, and Kami LeMonds, Inside SQL Server 2005 Tools, 0-321-39796-7 (Addison-Wesley)

Jeffrey R. Shapiro and Marcin Policht, Building High Availability Windows Server™ 2003 Solutions, 0-321-22878-2 (Addison-Wesley)

William R. Vaughn with Peter Blackburn, Hitchhiker's Guide to Visual Studio and SQL Server: Best Practice Architectures and Examples, 0-321-24362-5 (Addison-Wesley)

Buck Woody, Administrator's Guide to SQL Server 2005, 0-321-39797-5 (Addison-Wesley)

For more information please go to www.awprofessional.com/msserverseries


Foreword

In 1992, Microsoft first launched Microsoft Access Version 1.0 as a competitor for FoxPro (which it bought), Btrieve, Paradox, PowerBuilder, and DBASE. Many think that this was Microsoft's first foray into the world of database engines. It wasn'tMicrosoft introduced its first DBMS system in 1988 when it joined with Sybase and Ashton-Tate to launch its own version of SQL Server. While designed for use with OS/2, Microsoft quickly moved to leverage the considerable work they had been doing on their new NT operating system. This made the Microsoft version faster, smaller, and more competitive. It would eventually become the market leader in a very competitive DBMS industry.

Microsoft's first "RAD" development tool for Windows was created in 1991 with the introduction of Visual Basic. While Visual Basic took three versions to integrate data access into its user interface, Visual Basic was not the first Microsoft BASIC language to access SQL Server. That honor goes to Quick Basic after a Microsoft developer working on the SQL Server team built his own data access interface to make it easier to access SQL Server's only published API interface, DB-Library. Again, Visual Basic has evolved into the world-class Visual Basic 6.0 and more recently Visual Basic .NET languages. Much of the success of SQL Server can be directly tied to the ability of developers at all skill levels to create data access applications that leverage SQL Server's power, scalability, and flexibility.

I joined Microsoft in 1988 where I had the pleasure of serving as the first program manager for SQL Server. Our teams worked with Sybase early on and eventually helped create the Open Database Connectivity (ODBC) standard. Later in my career, we helped bring together the Visual Studio family of developer tools and drive the Win32 API, customer requirements, and product specification for the Microsoft Windows NT operating system. Since then, I have held many different roles at Microsoft and most recently have taken on the responsibility of Senior Vice President of the Server and Tools business unit that encompasses both SQL Server and the development languages.

During these years, we all witnessed SQL Server's market share grow steadily from an "all other" slice to over 50% market share and Visual Basic grow to be the most popular language on the planet. However, one problem Microsoft faced throughout this evolutionary period was documentation, training, and user-education. Since the SQL Server documentation team focused its attention primarily on SQL Server, and the Visual Studio User Education team focused primarily on user interface, tools, local data, and language issues, a gap was created that only someone familiar with both disciplines could tackle.

From the very earliest versions of SQL Server, while working for Microsoft University, Bill Vaughn wrote SQL Server courseware, managed and mentored SQL Server trainers, and wrote books and articles. This entire body of work was focused on helping developers understand how SQL Server and development languages like Visual Basic and Visual Basic .NET can be used to best leverage the power of SQL Server. His efforts to tie SQL Server and Visual Basic together were continued while he worked for the Visual Basic user education team and since retired from Microsoft in 2000. I first met Bill in 1988, and he was often a thorn in my team's side as he lobbied for more emphasis on SQL Server and IT data issues and better access to SQL Server's sophisticated features. It was clear that Bill was not afraid to express his opinions on how the languages and tools should be adapted to handle what he called "IT data." I guess this is because Bill had worked predominantly with very large databases at Electronic Data Systems and earlier in his 34-year career. Since Bill came in contact with so many IT developers while at MSU, many of his ideas were tempered with real-world field experiences offered by his students and conference attendees. No, we don't always like what Bill says in his books, but we respect his point of view. When Bill sees an issue with SQL Server, or the languages, or how they interact, he lets us know through his books and articles. Clearly, Bill is an independent thinker, and while he certainly knows and endorses Microsoft technology, he's quick to point out what works, what doesn't, and how developers can work around the issues.

Over the last 22 years, Bill has contributed a significant amount of quality data access documentation and training. His efforts have also yielded (now) seven editions of his well-respected Hitchhiker's Guides, as well as three additional ADO and ADO.NET booksmany of which have been translated into nearly a dozen languages. Developers all over the world have learned to depend on and respect his vast body of work.

This most recent book has been a monumental effort for Bill. Working almost three years on this edition, Bill has drawn on many respected industry sources as well as many of my team members here at Microsoft. He's attended and actively participated in any number of professional conferences and been invited to our own Microsoft software design reviews where Program Managers gather feedback and suggestions from industry experts. His input was invariably on-target for the developer community he represents.

We're extremely excited by the success of .NET and, with 2.0, the opportunity for Visual Studio continues. One of the reasons I can heartily endorse Bill's latest book is that it simply makes it easier for developers at all skill levels to get up to speed quickly on .NET technology using prose that any of them can understand. I support his efforts and recommend this text strongly to developers.

Bob Muglia

Senior Vice President of the Server and Tools Business (STB), a part of the Platforms & Services Division at Microsoft


Acknowledgments

Peter Blackburn

I've now known Peter for many years. During that time, he's been my technical rock. He keeps me on track in any number of areasselflessly providing his time and expertise whenever and wherever he can. He's a close friend, far beyond being a technical mentor. Peter's greatest qualities are his insight, honesty, and integrity. He stridently challenges all of usdevelopers and citizens aliketo protect ourselves against those who would steal, pilfer, and vandalize our data, intellectual property, and the things we've worked so hard to achieve. He's shown me cracks in the security strategies under discussion and how easy it can be to get inbut his scruples prevent him from capitalizing on those flaws. I'm just glad he's on "our" side. As a result, he's been hired by some of the largest firms in the world to review their systems to help fill those chinks in their armor. I don't know that I could have written this book without Peter. He has helped me make sure the technical details are right and kept me on track with some of the uglier aspects of Visual Studio and SQL Serverlike C# and (ugh) XML. Peter, thanks.

Microsoft Development Teams

I was very privileged throughout the writing of this book to have a direct access to key members of the development teams at Microsoft. Yes, some of this access is due to my status as a Microsoft MVP, but some of it is because I consider many of these people friendsI've worked with them for many years before and after my "retirement." In some ways, this relationship has been symbioticas I point out issues, the team either tells me they'll fix it, they show me why it's not an issue, or they put it on the stack of issues to fix one day. I've seen this interaction result in better products and certainly enabled me to produce a book of much higher quality than would otherwise be possible.

Sahil Malik, Winsmarts author, trainer, and consultant
Steve Lasker, Microsoft PM/Technical Lead for the UiFX and Services Team
Brian Randall, MCW Technologies Sr. consultant
Robert Green, MCW Technologies Sr. consultant
David Sceppa, Microsoft PM
Pablo Castro, ADO.NET Technical Lead
Billy Hollis, author and consultant

When the book was ready for technical review, many of the people on the Microsoft campus were busy grinding out the next versions and service packs of Visual Studio and SQL Server. No, I don't think this effort was simply an attempt to make my latest book obsoleteat least I hope not. The same was true for many others who still provided some of their time to provide feedback and encouragement. I appreciate the help they were able to provide over the two years it took to write this book.

The Addison Wesley Team

Of course, the book would not have been done at all without you folks. Thanks for putting up with me and my strange ideas and unreasonable demands. Included in this list are John Fuller, John Wait, Karen Gettman, Curt Johnson, Elizabeth Hurley Peterson, Andy Beaster, Eric Garulay, Neil Rowe, Mark Renfrow, and a host of others. Thanks, all. And thanks for getting the book through the production cycle in record time. Please strike this last line if we don't make the November 6th date. ;)

Marilyn Vaughn

Marilyn has (again) helped keep the grandkids, cats, and salesmen out of my office while I wrote this final book. She was there to listen to my incomprehensible accounts of how the day was going and to nod at the right time. She was there to keep me out of trouble when I went on the road and only collapsed once while shopping on the strip in Las Vegas. Thanks.

Victoria Ballard (George) and Chrissy Vaughn (Fred)

Ladies, you are my finest achievement. Since you have your own lives, you can't take the time to review or copyedit or make sure I'm not offending the people in some small town or major country anymore. But the lessons you've taught me over the last few decades are sewn into the fabric of all I write. I am very proud of both of you. Thanks.


About the Authors

WILLIAM R. VAUGHN has worked in the computer industry since 1972. He holds a Bachelor's degree in Computer Science from Mary Hardin-Baylor and a Master's degree in Interdisciplinary Studies from the University of Texas. He's also been awarded an Honorary PhD. from the University of Advancing Computer Technology in Tempe Arizona. Bill spent his early years working in the mainframe data processing industry and transitioned to the personal computer side in the late 70s. Along the way, he worked for the Texas DPS Narcotics Service, EDS (where he was recruited by and worked for Ross Perot), at Mostek/United Technologies, Challenge Systems, Digital Research, and CPT Corporation. After having surfed the PC industry for many years, he began his Microsoft years in 1986 working for the Windows developer liaison team at Microsoft in 1986. For the next 14 years, he worked in various divisions at Microsoft including Microsoft University, the Visual Basic documentation and Visual Studio marketing and Internal Technical Education teams before retiring in 2000 to form his own company, Beta V Corporation. Bill has written seven editions of the popular Hitchhiker's Guide (the 4th, 5th and 6th published by Microsoft Press) and books for APress, including the bestseller ADO and ADO.NET Examples and Best Practices. Peter Blackburn and Bill also coauthored the Hitchhiker's Guide to SQL Server 2000 Reporting Services (Addison-Wesley). His latest work is Hitchhiker's Guide to SQL Server Everywherehis first e-Book. He writes lead articles for SQL Server Magazine where he is a contributing editor, MSDN, and others. Bill also writes a bi-weekly editorial for Processor.COM magazine. Bill is a top-rated speaker at conferences worldwide, including keynotes and sessions at TechEd, DevWeek, Dev Connections, SQL Connections, VBUG, and many others. He is a Microsoft MVP and a member of the INETA Speaker's Bureau.

PETER BLACKBURN has worked in the computer industry since 1981. He studied computer science at Cambridge University, England from where he holds an M.A. degree. Through his consulting company, Boost Data Limited, Peter has designed, built, and implemented small- and large-scale database systems including reporting systems based on SQL Server. He has contributed countless hours to the development and honing of Microsoft Reporting Services and remains directly involved with the development team. He is also a Microsoft MVP. He supports developers through newsgroups and beta programs, and trains teams of developers working with SQL Server and Reporting Services. Having worked closely with .NET since before the launch, Peter and Boost Data have ensured the technical accuracy of many leading .NET books. Peter lives in the United Kingdom and travels all over the world to consult, speak, and provide support for his fellow developers, fellow MVPs, his writing team, and Microsoft itself.


Introduction

Welcome. It's been a while since I wrote my last Hitchhiker's Guide that focuses on application design and development. The previous book in this series, Hitchhiker's Guide to Visual Basic and SQL Server, 6th Edition, was first published by Microsoft Press in 1998while I was still working for Microsoft's Internal Technical Education group. Visual Basic 6.0 had just hit the streets, and SQL Server 7.0 was about to be launched. The introduction from the sixth edition observed that a lot of important changes had been made since the previous editionsignificant changes to Visual Basic, ADO, SQL Server, and the tools and techniques to connect them. I didn't think that the technology could change much more radically than introducing a fully compiled Visual Basic and a new version of a fairly sophisticated data access interface (COM-based ADO) to drive it. I was wrong.

Today, given the current state of SQL Server and what Microsoft calls "Visual Basic," saying that a lot has changed over the last seven years would be a gross understatement. Just before I retired from Microsoft in 2000, Microsoft walked away from its COM-based version of Visual Basic (6.0) as it introduced its successor: Visual Basic .NET built on the new .NET Frameworkthe foundation for Vista. At the time, it was clear that these changes would require a cover-to-cover rewrite of my Hitchhiker's Guide, but I was not in a position to take on a project of this magnitudenot until the dust had settled on .NET. Instead, I wrote two more focused books for Apress: ADO and ADO.NET Examples and Best Practices for Visual Basic Programmers, for Visual Basic 6.0 developers transitioning to Visual Basic .NET; and (with Peter Blackburn) a C# version: ADO.NET Examples and Best Practices for C# Programmers. Neither of these ADO-focused books really addressed many of the fundamental architectural concepts and techniques included in previous editions of the Hitchhiker's Guide.

Along the way, Peter Blackburn came on board and was instrumental in getting these two Apress books to market and subsequently convinced me to write a more focused book on his favorite topicSQL Server Reporting Services. Since the next version of Visual Studio (Whidbey) was not due for another year, it seemed like a good idea (at the time). This project was expected to generate a 200-page book in about three months. In actuality, it took over a year to grind out, but Hitchhiker's Guide to SQL Server 2000 Reporting Services (Addison-Wesley) is a 780-page tome that's the most respected book on SQL Server Reporting Services yet published. Peter did the bulk of the research and initial writing, and provided significant (and greatly appreciated) technical guidance and logistical support for the seventh edition.


How Did I Get to This Point?

What's so special about the Hitchhiker's Guide series? Well, perhaps a bit more history will help clear this up. In the late 1980s, I worked for Microsoft University (MSU) writing and delivering courseware on any number of now-obsolete technical subjects like OS/2 and Quick Basic. When Microsoft established a partnership with Sybase to co-market and develop an OS/2 version of SQL Server, MSU and I were recruited to build courses to teach developers relational database design, TSQL programming, DBLibrary programming, and other related subjects. I managed and contributed courseware to the group of trainers who wrote and delivered this courseware for almost five yearsoften three to five days a week, three weeks a month. Along the way, we all learned a great deal about SQL Server and best practices.

When MSU folded, I wrote my first Hitchhiker's Guide: Hitchhiker's Guide to VBSQL. No, the title had nothing to do with Douglas Adam's bookit was to be a "tour guide" like Europe on $5 a Day for SQL Server front-end Visual Basic developers. The book's manuscript helped me get a writer's slot on the Visual Basic documentation team. The first edition was a folksy tongue-in-cheek treatment of Visual Basic (1.0), DBLibrary, and SQL Server, and how they could interconnectthe tips and techniques were drawn from the courseware I had written and my interaction with SQL Server developers and MSU students. Using some pretty silly and irreverent examples and prose, I was able to make a fairly dry topic humorous and very readable, while still getting the heavy technical content across. I took the first edition to Microsoft Press for publication, but (strangely enough) they wanted a Power Builder book, not one on a (then) obscure DBMS like SQL Server. As a result, I published the first three editions myself and eventually sold several thousand copies out of my homethese first three editions made their way to customers and bookstores all over the world. I even sold a single copy to a fella in Hong Kong who made it very popular in Chinatoo bad I didn't get anything back from these copies.

Each subsequent edition of the Hitchhiker's Guide leveraged the previous as I expanded my understanding of Visual Basic and SQL Server and as these products evolved and matured. Each edition focused on the latest Visual Basic and SQL Server interfaces and features, and each became the definitive source for information on ODBC, DAO, ODBC Direct, ADO, and the Visual Basic tools used to access these data access interfaces. I like to think that the book itself helped focus Microsoft and the data access teams on addressing many real-world issues that my students, customers, and readers constantly relayed to me. As I created new solutions to common problems, I would fold these into the next edition. Over the years, I've spent considerable time interacting with the developer communityat conferences, via newsgroups, and in training sessions. If you've met me and made a good suggestion for a better way to handle a data access issue, it has probably ended up in the book somewhere.

Eventually, Hitchhiker's Guide's popularity encouraged Microsoft Press to ask to publish the fourth edition. After we came to an understandingI would maintain full editorial control (the jokes and Microsoft barbs had to stay)I relented and the soylent-green fourth edition was published. Microsoft Press subsequently published the updated fifth and sixth editions over the next few years; it was subsequently translated into at least eight languages, including Japanese, Spanish, French, German, Hungarian, Italian, Korean, and Chinese.


The 7th Edition's Scope

The Hitchhiker's Guide to Visual Studio and SQL Server, 7th Edition, is intended to provide a broad treatment of Visual Studio and SQL Server, how they interact, and how developers can build professional applications using these tools and engines. In case you didn't pick up on this from this introduction, that's been my specialty for some time. When working at Microsoft University and over the last 20 years interacting with developers and other resources, I accumulated a wealth of information that's not found in the documentation. When MSU folded and I moved to the Visual Basic documentation team, I was "permitted" to document Visual Basic but not SQL Serverthe documentation had to be "generic" and apply equally to all database backends. All of my books have filled this gap between the language (Visual Basic) and the various versions of SQL Server.

The seventh edition of the Hitchhiker's Guide takes an even broader view of data access development when compared to previous editions. That's because Visual Basic. NET is not the Visual Basic language discussed in the earlier editionsthus the change in the title to Hitchhiker's Guide to Visual Studio and SQL Server. Today, RAD development is as much about Visual Studio as it is about the development language. That's because Visual Studio generates a significant amount of code in the selected language and provides so many tools, wizards, controls, classes, and interfaces that are common to all of the languages it hosts. As a result, the 7th edition spends considerable time discussing how the Visual Studio toolset can be, should be, and should not be used to create professional applications.

SQL Server has also introduced an entirely new paradigm. Server-side executables can now be written in Visual Basic .NET or C#. This means stored procedures, functions, triggers, user-defined types, and aggregates can all be written using Visual Studio and a familiar CLR language. Here in the seventh edition, I spend a long time discussing how and when it makes sense to design and implement each of thesewell beyond the scope of previous editions of the Hitchhiker's Guide. You'll find the entire book is focused on connecting to and running queries on SQL Servernot Oracle, DB2, or JET. Sure, developers who need to access other DBMS platforms have praised my previous books because the fundamental principles and best practices I describe often (usually) apply to other DBMS platforms as well.

In any case, I don't want to dissuade developers from buying the book just because "Visual Basic" or "C#" is on the cover; 99.44% of the book's content applies equally to C# and Visual Basic .NET, although most of the examples are provided in Visual Basic .NET. I had originally planned to provide parallel C# examples in addition to Visual Basic .NET, but I ran out of time and bandwidth. I expect most C# developers can easily translate Visual Basic .NET code to C#. But the same can't be said for a portion of Visual Basic .NET developersespecially those transitioning from Visual Basic 6.0. I also did not include a SQL Server version number in the title, as the discussions apply to any post-SQL Server 2000 version, including SQL Server 2005 and its service packs. Given Microsoft's stated intent to update its products every year or after every lawsuit, I needed to do what I could to keep the title (and content) from getting stale before its time.


Keeping Current

In all of my books, I've tried to provide up-to-the-moment information, but this is really a daunting task, as Microsoft is constantly updating, fixing, and tuning the released products. It's as if they don't really have a long-term plan of how we're supposed to write our applications. What's perfectly suitable today is passé, discouraged or dropped, or hidden or deprecated tomorrow. I don't see an end to this constant cycle of early release, repair, and planned obsolescence as Microsoft attempts to maximize their profits by forcing development shops (and users) to buy their latest whiz-bang versions. It seems there is never enough time to build a product that include the features ordinary developers need and (sadly) never enough time to fix it once it ships.

As an author, I'm also faced with a dilemma. At this point in time (April 2006), I have over 750 pages of original content sitting on my hard-drive. I can't show it to anyone until the book gets into print. Given that the useful lifetime of technical content is approaching the duration of a phosphorus flashbulb, we (the publishers and I) have to figure out a new way to get current content into your hands more quickly. The other problem I'm faced with is scope. Frankly, I would have liked to include far more in-depth information on many other related subjects, like ASP development issues, security, deployment, SQL Trace, the profiler, and others. I would also like to discuss the new DLINQ and other emerging (and experimental) data access technologies to be released with the next version of Visual Studio (Orcas). I found that this was simply impossible to accomplish when working virtually alone.

To address these problems, I'm planning to write and publish (one way or another) a set of EBooks on related subjectsyou'll see references to these as you read. Ironically, the first EBook might be available before the first printed copiesit seems to take about five months to get a book from draft to paper nowadaystime enough for Microsoft to reinvent the entire tools suite. EBooks are generally shorter than paper books (Addison-Wesley even calls them "shorts")usually just a single chapter or sectionso they are far easier to write, edit, and distribute. Unlike paper books, EBooks can also contain full-color illustrations and provide links to online animations, Camtasia videos, or other supplementary content.

Paper-based book and magazine publishers are trying to figure out how to best market and sell EBooks, but it remains to be seen if they have the vision and courage to transition away from their traditional printing and distribution paradigms. The problem we all face with EBooks is piracy. All too often, as soon as an EBook is published, those who would cheat the system put the content on the Internet. This robs authors and publishers of their motivation to publish, and everyone suffers as a result. Several months' work can be flushed away in an instant. Until this problem is solved, I don't expect many publishers will fully explore this unchartered territory. I'm going to experiment with a few EBooks to see how well they are received and whether the rumors about piracy are true. My first will be on SQL Server Everywhereif it's still called that by the time you read this.


Is This Book Written For You?

I'm often asked if my books are a good match for the skills and interests, and needs and focus of one group of developers or another. While I would like to say, "It's for anyone who can read," that's not really true, as you might expect. I can say the book is for:

  • Software developers, architects, and their managers who specify, design, code, assemble, test, deploy, support, update, and live with business applications that manage dataespecially SQL Server data.

  • Developers with some experience in Visual Basic 6.0, Visual Basic .NET, or C#; or those developers skilled in other languages who want to understand these languages and how to use them to create data access applications.

  • Anyone who wants to leverage the latest and ever-evolving Visual Studio and SQL Server feature setlearning what works, what doesn't, and why.

  • Consultants and mentors who want to learn how to best support their SQL Server customers by creating quick applications more quickly.

  • Those who want to try to learn how to best use SQL Server for the first time, as well as those who have experience with SQL Server and its versions and variations.

  • Any of the previously mentioned people who wants to know how and (more importantly) when to leverage the power and potential of SQL Server 2005 CLR executables or SQL Server Reporting Services through the new Report Viewer control.

  • Students, teachers, trainers, and courseware designers who want to gain or pass on useful skills to be competitive in the software-development business.

  • It's written so I can simply point to a place where anyone anywhere on the planet can get another point of view on how to develop data access applications using Visual Studio and SQL Server.

  • It's also written for me. It's my way of passing on what I've learned by experience and from my readers, my session attendees, and those folks whose questions I've answered over the years.

  • It's also for Alberta Lee, my 81-year-old mother-in-law, as she puts a copy of the book on the mantel to show her friends.


Getting Help and Support

Look for links to my EBooks on the seventh edition's support website: www.hitchhikerguides.net. A six-month paid subscription to this site is included with the price of this book. All I ask is that you register with the site using the "scratch-and-sniff" number found just inside the front cover of the book. This, along with a valid email address, will permit you to gain access to the "premium content" area that includes updates, errata, and reduced-cost access to my growing number of EBook offerings, as well as the ability to participate in the reader's forum section of the site. I encourage you to use the site to provide feedback, ideas for other content, and support for other readers. I hope this helps.

William R. Vaughn

President, Beta V Corporation


Chapter 1. Exploring Application Architectures

    Introduction

    Choosing the "Right" Architecture

    Understanding Your Toolset

    Recognizing Application Design Constraints

    Choosing the Right Data Access Interface

    Choosing the Right Database Management System

    Understanding Basic Data Access Architectures

    Summary


Introduction

A professor at MIT is quoted as saying, "Design is what you do when you don't [yet] know what you are doing."[1] This is very trueeven for application architects. It seems that so many of us depend on the wisdom of others to design their systems that we fail to learn the fundamentals of system design. As a result, the end product can scale, run efficiently, and be maintained for decades to comecome what may.

[1] George Stiny, Professor of Architecture, Massachusetts Institute of Technology, 21 June 2002.

IMHO

"Too many of the designs I see lack the solid foundations needed to have them last beyond the next strong wind."[2]

[2] William Vaughn, President of Beta V Corporation, 22 July 2006.

Before getting started on a tour of Visual Studio and SQL Server 2005, I feel a need to lay some groundwork. In my previous books[3], it was assumed that the reader had a firm understanding of how SQL Server works, knew how to write T-SQL, and knew how to build a normalized relational database. Since then, I've seen too many questions that could be simply answered by a better understanding of those fundamental tenants. To make it easier for you to understand why one architecture or technique is suggested over another, let's start by providing some foundational material that should help make your choice of architecture and how you build your applications easier. Of course, if you think that you're up to speed on all of these issues, feel free to skip on down.

[3] I've written at least 11 booksnot counting the volumes I wrote for Microsoft and other companies over the years.


Choosing the "Right" Architecture

When presented with an application challenge, we often have to choose a workable approach from myriad alternatives to implement the design. As when building a new house, the choices you make determine how well it holds up and meets your customer's or your personal needs when it's used or abused, or when a hurricane blows through. Your design and implementation (the way you build your house) also determine how easily you can add a new wing or simply remodel the kitchen. Your design and implementation also determine whether or not the building inspector will approve your home for occupancy or force you to go back and add bathrooms. Yes, there are always constraints. Just as building codes, the homebuilder's budget, and the site chosen dictate what kind of house you build, software architects and developers are limited as to budget, schedule, integration with existing applications, databases or operating systems, and what the boss's nephew thinks the program should do. Experience tells us that "people" issues often play a pivotal role in the choice of architecture. As architects, mentors, and consultants, we all must concern ourselves with training (or retraining), installation and deployment, and licensing costs (assuming your company actually pays for its software), as well as myriad development issues. We've all seen situations where the company was dead-set against one or more architectures long before we arrived and no amount of logic would shake them from this opinion. Perhaps, the company has built their system up over several decades and, due to its immensity, can't possibly change it. While these situations are not always dire, they can significantly impact the success of your working relationship and your ability to provide a "best practice" solution.

Sometimes (fairly often, in my experience), you'll be called on to modify, repair, or resurrect an existing applicationperhaps one you didn't design or write and all too often based on a design that you can't endorse. Perhaps the company is looking for validation for an architectural approach or some way to provide independent credibility for a decision made about the application.

Office politics is another factor that often plays a big role in application design. If I've learned anything over the last thirty-some-odd years it's that people are very sensitive about their creative ideas. Criticizing their design is tantamount to saying their kid's face resembles a pig's butt (even though it does). They also tend to be defensive about designs sold to upper management or suggested by someone they respect. If the design fails or is somehow flawed, someone's career might be in jeopardy. When you approach an existing design, consider who created it, its history, and the motivation behind that approach before you add your offhand or well-considered comments.

IMHO

Criticizing a customer's design is like telling them their kid's face resembles a pig's butt.

I lost count of the number of times someone has asked about a problem with their application only to tell us that they didn't really have a choice as to a better (best practice) alternativeit's not their design. I expect many of you have worked with customers who tell you how to design the system down to the finest detail but blame you when it does not work as they expected. Doctors have the same problem when a patient who's been watching the 6 PM news decides that they have every disease the pill-pushers are advertising. It's hard to know what to do in these casesthere is no simple answer. Do we tell our customers that their design won't work, it's somewhat "challenged," or do we carefully outline specific problems and areas of concern? We could just walk away and lose the account to someone who needs the job or doesn't know better.

My approach has been to try to prevent these problems in the first place by mentoring the customer as the specification is written and work with their team to better understand the issues, challenges, and dependencies before the design is set in stone. If I arrive on the scene too late or they insist on a challenged design, I insist on documenting our concerns and either walk away or do the best I can given the constraints. Frankly, sometimes it's best for everyone if I turn down projects with a doomed design. It's like trying to repair an engine that melted down to a puddle trying to deliver 5 tons of bricks with a ¼-ton Toyota pickup. Sometimes, it's cheaper and easier (in the long run) to get a new truck that's better designed for the load. As we've all seen, even the most prestigious leaders have trouble knowing when they're in over their head and admitting they made a mistake in judgment.

On the other hand, perhaps our skills are not a good match for the project. Consider this: When surgeons approach a patient's problem, they try to solve it with a scalpel because that's the tool they know best. When chiropractors approach the same problem, they try to solve it with an "adjustment," just as a podiatrist would use a lift or an orthopedic shoe. In the same way, when a programmer that's worked exclusively with JET/Access databases approaches a problem, they're likely to design the solution with JETeven though it's rarely the best fit. In the same way, ASP developers tend to choose browser-based solutions just as Windows Forms developers tend to choose a client/server approach. Unfortunately, too many developers choose an approach because it's new or sexy or has "XML" in the name instead of the approach that best fits their customer's requirements. In our world of specialization, consultants and developers often have to know when their skills and specialties apply to the problem at hand and when they don't. If you encounter problems that can't be solved with your skillset, you need to get more training, do more research to add more skills and talents to your toolbox, refer the problem to a specialistor know when to back away. I assume that's why some of you are reading this book.


Understanding Your Toolset

Today, ADO.NET is the mechanism applications of all kindsregardless of their architectureuse to communicate with SQL Server. Yes, because Visual Studio generates much of this code for you (if you choose that route), the low-level ADO.NET commands are often hidden from view like the gears in your car's transmission. ADO.NET is certainly one of your most frequently used data access development tools. I'm convinced that if you don't know ADO.NET and how it works, your ability to design and build successful, long-lasting, and well-performing applications will be very limited. By leveraging its features and avoiding its limitations, one can more easily build these successful applications. ADO.NET is not new. It was not invented overnight (as some of the early SQL Server data access interfaces appeared to be). Its sophistication and functionality lie in not just what it does, but what it no longer does. Since many of you are coming to this book from COM-based ADO (what I call ADO classic, or ADOc), its important to know how we (you and I) got to this point.

A Brief History of ADO and ADO.NET

In the early days of SQL Server (when Ashton-Tate sold about four copies of SQL Server 1.0 circa 1989), the only way to write a program to access the server was to write a C program that used DB-Library (DBLib)SQL Server's first data access interfaceit can still be used today, but with considerable difficulty. SQL Server 2005 won't expose anything but basic functionality when accessed with DBLib. That's because Microsoft has not really kept it updated. They want you to use their new object-based interfaces and steer clear of low-level API-based function calls.

DBLib API-based functions accept parameters that build TDS[4] packets to open a connection to SQL Server akin to today's ADO connection. Using these API calls directly from an application was always a bit trickyif something went wrong, your customer's system could "blue-screen." If your application worked, it was very fast, and since there was virtually no client-side work being done by DBLib, much of the logic was written in TSQL server-sided procedures. In years past, many folks (and I) built applications using fairly simple paradigms where the data access interface was light, fast, and relatively simple. In those days, emphasis was on server-side intelligence leveraging the power of server-side stored procedures, rules, and triggers. Many of those applications are still in use todayand so is this "client/server" architecture, albeit no longer implemented with DBLib.

[4] TDS: Tabular Data Streamthe proprietary protocol used to communiucate with SQL Server.

Jet and ODBC Are Born

As SQL Server matured, so did Microsoft's attempt to refine and broaden the data access interfaces. Since SQL Server had virtually no market share, it was important to create an API interface to access other popular data sources in the consumer and business markets. Remember, this was before Visual Basic existed, and Quick BASIC was not seen as a "real" language, so C languagebased API interfaces were about all we had to work withat least, if we wanted to use SQL Server.

Microsoft took three paths at this point in time. First, it started the process of finding a "consumer" or small-business DBMS system to compete with Dbase, Paradox, Btrieve, FoxPro, and the like. Next, Microsoft started development of a new DBMS built on their "Omega" data engine, which would eventually become what I dubbed the "Joint Engine Technology," or "JET," ISAM[5] DBMS. Third, Microsoft created its first one-size-fits-all (OSFA) data access interface, dubbed Open DataBase Connectivity (ODBC), based on the SQL Access Group Call Level Interface (CLI), in 1992. It was destined to be the first of many "generic" data access interfaces they would create over the next decade. ODBC used "drivers" that translated low-level ODBC API calls to the proprietary interfaces exposed by the various DBMS engines. Eventually, hundreds of ODBC drivers were written and virtually every type of relational database could be accessed from this OSFA interface. While this "translation layer" approach slowed down the interface and sometimes ignored special features, it was widely accepted as a standard interface. The problem was that the DLLs used to hook up applications to ODBC were fragile. They had to be used in matched sets, and with so many drivers floating around (sometimes many for the same DBMS), applications sometimes crashed and brought the OS down with it. Remember, this was before COM came on the scene to "save the day" (and make matters worse).

[5] ISAM: Index Sequential Access Method. This class of DBMS is characterized by use of primary-key indexes to access data in flat files.

On the first front, Microsoft eventually bought FoxPro and immediately started to incorporate its innovative technology in its new JET engine. JET found a home in Microsoft Access and behind the scenes in Exchange (the mail engine). About the same time, ODBC gained acceptance, so the JET team tried to assimilate that technology as well. Unfortunately, it seems the JET team did not fully understand (or care about) SQL Server or large DBMS technologythe implementation left a lot to be desired. Performance was, well, terrible, and there was little support for stored procedures, security, multiple resultsets, and many other features that made SQL Server more than just a data repository.

On the language front, a developer in the SQL Server team went home one weekend and wrote his own interface to DBLib for Quick Basic that he (and I) dubbed "VBSQL." At this point in time (around 1992), I wrote the first Hitchhiker's Guide[6] about this technology as it expanded to incorporate the new Visual Basic. Microsoft missed their chance to get a head-start on ODBC connectivity when it failed to incorporate the ODBC API into Visual Basic 2.0. When Visual Basic 3.0 was launched (it was delayed to let the JET team get their engine ready), it incorporated a crude version (by today's standards) of the JET database engine and an even more crude interface to ODBC that could access SQL Server (just barely).

[6] Hitchhiker's Guide to VBSQL (Beta V Corporation).

Data Access Objects (DAO) Is Born

In those days, Microsoft Access and Visual Basic 3 used Data Access Objects (DAO) to access JET. DAO was developed for the Microsoft Office team, so it followed the tiered "hierarchical object" approach used in Excel. This made it less efficient, but it was the only way to access JETthere was no published JET API interface. DAO also made a pretty sad interface to SQL Server, as it barely permitted execution of stored procedures and made some fatal mistakes when the stored procedure returned more than one rowset (as they often did). In those days, DAO basically treated SQL Server as a file server.

Unfortunately, ODBC fell by the wayside as far as Visual Basic was concerned, although it was (and is) a far more efficient way to access serious DBMS systems like SQL Server and its arch rival, Oracle. This made VBSQL far more popular and helped developers adopt SQL Server as a cheaper alternative to Oracle, DB2, and Informix. Meanwhile, back in the Visual Basic group at Microsoft, Dave Sterns, Drew Fetcher, and I devised an independent plan to use ODBC to bypass JET. The idea was to access SQL Server via a more developer-friendly Visual Basiccentric data access interface, so Microsoft's Visual Basic development team implemented what became known as Remote Data Objects (RDO). According to the initial specification, RDO was supposed to be a light "wrapper" interface to ODBC. Unfortunately, it was implemented using the clumsy DAO object model (to make writing the documentation easier and less intimidating for DAO developers), so it got pretty bloated. As kludgey as it was, RDO easily outperformed DAO when accessing SQL Server, and it exposed stored procedures like never beforeeven complex resultsets where handled correctly. It even had the ability to retry timed-out commandsa feature that was never brought forward past RDO. I can't help but think that RDO helped SQL Server become a winner with many DBMS decision-makers. It made writing front-ends for SQL Server far easier, faster, and more stable than using JET or DBLib.

OLE DB Is Spawned

But Microsoft was not done. Because of one somewhat deluded (IMHO) but vociferous product manager, Microsoft eventually launched its most generic and most ambitious OSFA interface to date: OLE DB. This "Object Linking and Embedding" information interface was not restricted to accessing just relational databases like ODBCit could access any source of information, including flat files, the Active Directory, 80-column cards, refrigerator magnets, or any other information source, including (incidentally) relational databases. As a result, Microsoft put yet another (highly complex) layer between the low-level TDS (and other native protocols) and the DBMS engines developers were trying to query.

Another (not so) brilliant move was to create OLE DB in C++ using complex interfaces that could not be accessed directly from Visual Basicthe most popular and widely used language in the world (at the time). It seems that Dumb and Dumber were at the tiller. This is when COM was king and COM-based ADO was the answer to the Visual Basic developer's dilemma and, so it appeared, all of the world's problems. From its inception, ADO was innovative but problematic. While it was a far better and richer interface to SQL Server (it followed many of the innovations designed into RDO), it was buggy and suffered from radical and sometimes violent fits of temper as it transitioned from version to version. It seemed that every time Microsoft "fixed" ADO, it broke a few scattered applications or every ADO application in the world. Developers became more and more hesitant to install updated ("new and improved") versions of ADO (included in what was called the "MDAC stack"), for fear that they would (again) break existing applications. This syndrome was eventually to be called "DLL Hell." It was clear to some of us that COM had been stretched well beyond its envelope of usability.

ADO.NET Is Created

In response to the drumbeat of dissent from angry and frustrated developers (and their customers), Microsoft started over (again). The .NET Framework was built to provide a radically new programming interface for Windows to be launched as "Longhorn" (what became Vista) sometime in this decade. This seemed a natural place to incorporate the newest data access interface. Worried that developers would not adopt this radically new programming model, Microsoft marketing (in its infinite wisdom) dubbed the new .NET language for Visual Basic developers "Visual Basic .NET" and hoped they would not notice that it was an entirely new language. They did the same thing with ADO, calling it "ADO.NET," although the new data access interface had little resemblance to its namesakeexcept the fonts.

ADO.NET got its start as the new .NET technology first took shape. Microsoft knew that they needed to replace the aging COM-based ADO and wanted to make it more universal, lighter, and faster. They also were enamored by the newest darling in the world of technology, XML, so they wanted ADO.NET objects to serialize to and be generated from XML. Originally, ADO.NET was called "XDO" (XML Data Objects) because of that emphasis. They stuck with XDO until very late in the development process, but the marketing team changed it (to my dismay) to ADO.NET. I wish they had kept XDO as the name, as it makes developers aware that it's an entirely new (and improved) data access interface and does not carry with it the baggage of COM-based ADO.

In its defense, ADO.NET takes an entirely new approach. Sure, it's an OSFA data access interface, but instead of trying to force developers to use generic interfaces, it's designed as a way to access the low-level protocols. For the first time since DBLib, Visual Studio developers can access TDS through a fairly thin layer of ADO.NET classes and .NET data access providers. Unlike COM-based ADO (ADOc), ADO.NET does not try to do much in "black boxes" of code over which developers had little controlat least, for the most part. It does not try to guess what kind of SQL is being executed; it does not try to build a Parameters collection for queries or construct UPDATE statements on the fly like ADOc (tries to do). This makes ADO.NET lighter, faster, but more code-intensive and, thus, harder to work withit means the developers have to write a lot of code or use wizards to build the code for them. That's where Visual Studio .NET comes in. Visual Studio includes a set of wizards that automatically generate code at design time that ADOc generated at runtimea radical departure. Unfortunately, Visual Studio's wizards still don't provide all the flexibility and features afforded by ADOc's black boxes. For example, in ADOc, one could set the Update Criteria property to determine what SQL was to be generated to manage update collisions. While ADOc offered at least four options, Visual Studio 2005 offers but twobut I'll get to that later (in Chapters 6, "Building Data Sources, DataSets, and TableAdapters," and 13, "Managing SQL Server CLR Executables").

ADO.NET still has not caught up with ADOc in a number of ways. It does not support any form of server-side cursors, nor does it support asynchronous connections, asynchronous rowset population, update concurrency management, value-set exceptions, and many other features. Yes, these features have been on the list to include in ADO.NET for over 6 years, and while they have tried, the ADO.NET development team has chosen to implement other new features instead. This simply means that those developers trying to convert from ADOc will have more issues to deal with, as there are no equivalent features in ADO.NET to implement.

What's a Client and What's a Server?

Before we go on, let's make a few terms clear. When an application opens a connection to SQL Server (or any DBMS data source), it's considered to be a "client." This "client" application can be a Windows forms application, a middle-tier XML Web Service, an ASP application, or an application on a hand-held PC connected to a milking machine. "Client/server" applications are considered to be those programs (like Windows forms applications) that open (sometimes persistent) connections and provide interactive user interfaces but are not ASP-based. That is, the application runs on an independent computer, separate from the system that hosts the DBMS engine (like SQL Server).

A "server" is the system that hosts SQL Server or other shared DBMS. Yes, a server can share files, and it's then dubbed a "file server." SQL Server originally got its name from the fact that it provides Structured Query Language (SQL) "services" to the network and runs as a Windows service on the host. Today, the term "SQL" has expanded well beyond its original meaning. Don't be confused by "SQL Server" Everywhereit's not a service-based DBMSit's file-based, as I'll discuss later.

In a nutshell, ADO.NET is Microsoft's newest data access interface. In Chapter 8, "Getting Started with ADO.NET," I'll talk about how ADO.NET works and how it's different than ADOc. This should help you transition from ADO classic, if that's on your agenda.


Recognizing Application Design Constraints

Each architectural approach discussed in this chapter and detailed in the book has its pros and conslike everything else. Of course, if you consider that your situation might need several applications to meet your needs or a customer's needs, you might have to choose a combination of several approaches. Just keep in mind that not all of these architectures work well togetherif at all. Each of these architectural alternatives deals with security, connectivity, data retrieval, data persistence, user interaction, and many other factors in different ways. Some architectures (like Web Services) have no user interface (UI) component at all. Some permit you to manage server connection state, and others suggest a "stateless" approach that permits a more scalable architecture. Being stateless can mean that you live on a ship sitting offshore because the feds are looking for you, or it can mean that your application does not maintain variables or data structures between invocations of your code. In this case, each time your stateless application is invoked is just like the first time. Sure, there are very few purely stateless applications. Depending on the architecture and what you implement, an application manages its state primarily on the server, on the client, or somewhere in between. In any case, when there's less state to manage, there's more time and resources to support more users. As these architectures are discussed and contrasted in the following chapters, I'll try to show the types of applications each architecture is best suited to handle.

Before you choose one architecture over another based on its scalability or performance characteristics, consider that the vast majority of data application implementations don't have to scale past a few dozen (20100) users. This means the techniques you use don't necessarily have to be that sophisticated or complex. Remember that scalable architectures are usually slower and less interactive, but more clients can be supported.

As you tour this chapter, I'll discuss common application design mistakes, inconsistencies, and outright blunders. Sure, some are more serious than others. For example, we've all seen developers build systems to fetch data from a DBMS Server, process it, and return it to the same DBMS Server. These folks often overlook the fact that it usually makes more sense to process the data directly on the server using a server-side procedure. This approach eliminates one of the most common performance bottlenecks: "round-trips" to and from the server as the data is streamed to and fromoving (too much) data to the client (and back). Sure, there are cases where it's hard to write a server-side procedure to perform some calculations (especially if you're not comfortable with T-SQL programming), but it might be possible to create a CLR server-side procedure to do the work more efficiently in a programming language you understand. I devote Chapters 14, "Creating and Managing Reports," and 15, "Summary and Wintry: Where We Are Now," to writing, testing, and deploying CLR-based executables.

What Makes "Good," "Better," and "Best" Designs?

What factors should be used to determine which architecture is better than another? Frankly, there are a virtually unlimited number of factors to deal with in the decision-making process. The key to success is discovering the critical factors that apply to your specific situation as you formalize your design (not as you implement it), while at the same time filtering out the factors that don't really impact the final productthe "noise." For example, if you build your design around MSDE (the "free" version of SQL Server 2000) or SQL Express (the "free" version of SQL Server 2005), your design will fail if the customer subsequently decides to store 10GB of Word documents and pictures in the databasethese free versions limit the database size to 2 or 4GB. As I've said, you also have to strike a careful balance with the constraints, such as what the customer wants (or thinks they want), the budget, the schedule, and other limited system resources, such as RAM, disk space, CPU, and LAN/WAN capacity.

IMHO

All too often, customers and managers don't know what they want until they don't get it.

Remember, one of the most important factors in a successful implementation is the human element. What skills are in place at the customer site and in your company to implement the design? Does the site have "real" developers or is it overflowing with "paradevelopers" who are not trained in computer science or the languages and tools needed to get the job done? Does your company have what it takes to get the job done? What about the end-usersthe people who will actually use the end-product application? Who are they? Are they random people out on the Internet or company employees? Do these folks need additional training to use your application or computers in general? If so, is this training written into the specification (and contract) or provided online in real time? Who is supposed to write and deliver this training? Will the end-user have sufficient rights to access to the LAN? Why design a connected application if the customer won't have access? Why develop a Windows Forms application if the customer's desktops are Mac-based or they're still running CP/M and 8-bit Challenge 2000 systems?

Let's go over a few goals that can help focus our search for the best architecture for your project. These approaches are designed from the beginning to:

  • Solve the customer's problem today and not be a source of additional problems tomorrow.

  • Make and keep the customer productive and competitive and out of your haireven while you're building a solution to their problems.

  • Be scalable within reasonable growth expectations. Not all applications need to support 10,000 (or even 1,000) users.

  • Build applications that perform quickly and efficiently enough to deal with the expected load, given expected growth.

  • Log and handle exceptions with the least amount of customer interaction or disruption.

  • Build applications that are implemented and maintained by the customer's development and support teams without constant need for outside experts to constantly tune, patch, maintain, and troubleshoot the systems.

  • Use development standards that don't require special training so that replacement developers can easily work with the existing code.

  • Function within the constraints of the customer's resourcesnot requiring the expense of additional systems, staff, or software licenses.

  • Integrate seamlessly with existing systemsor at least tolerate their existence.

  • Document the entire application and its support systems, including end-user instructions, as well as build, maintenance, deployment, and configuration instructions.

  • Incorporate security systems that "sufficiently" protect the customer's data and comply with all government regulationseven the ones you don't know about (yet).

  • Deal seamlessly with backup and disaster recovery issues.

  • Produce a workable solution on time, within budget, that includes rolling progress reports.

IMHO

In today's uncertain times, we all need to be building redundant and carefully protected systems.

What I'm suggesting here is an architecture and design with which both you and your company can be satisfied and where a developer can walk away in good conscience when the contract is done. That is, the customer is happy and secure with the implementation and doesn't need to call you on Saturday afternoon just as you push off from the dock for a cruiseeven when there is a crisis and it's not a design where you feel that you need to leave the country to avoid prosecution.

I cannot overemphasize the importance of adequate securitysecurity that's built into a system, not tacked on like plywood duct-taped to the sides of a Humvee. Because of more stringent government regulations and threats of prosecution (and because it's the right thing to do), most of us strive to design systems that don't make it easy for hackers, "evil-doers," Veteran's Administration employees, or anyone else who would compromise the data our applications are designed to store and protect. I (generally) never know what my customers store in their databasessometimes I don't want to know (or I should not know). I do know that the data should not be exposed to just anyone without the appropriate rights and credentials. For example, in the U.S., if your application deals with healthcare information (patient data), you're going to have to comply with HIPPA (a.k.a. HIPA) regulations. In other countries, there are similar, if not more stringent, security requirements. It's beyond the scope of this book to describe these laws and how to show you how to make sure your code complies with these laws. In any case, your application must be built around a DBMS engine capable of implementing these fairly rigid security requirements. This is another reason I can't recommend JET/Access databases for professional implementations where security is a factorit simply can't be adequately protected. While single-user JET databases can be protected (to some extent) by Access/JET security and the file system ACLs, these are not that hard to crackno, I'm not going to tell you how.

IMHO

I don't recommend JET/Access databases for any professional implementation where security is a factorit simply can't be adequately protected.

Designing Before Implementing

Ideally, when I sign a consulting or mentoring contract, I stipulate what's expected of the customer and what they expect of me. I also like to work up a pre-written specificationa detailed description of the application to be built. It's like the blueprints for a house that includes everything from a detailed drawing of the finished house to the electrical, plumbing, foundation, ventilation, and other detailsright down to the color of the carpet and the kind of toilet fixtures. Some customers know exactly what they want you to do and can provide detailed instructions. Unfortunately, we all have worked with customers that don't know what they want until they don't get it. All too often, customer demands are fairly vague. "Make it better" or "Make it faster, cooler, more competitive, easier to use; do it in the next two days; and be sure to use XML" are typical demands. This means that it's our job as consultants, developers, and mentors to help the customer take this first (big) step and figure out what he or she really wants, what the company wants, and what resources are available to make it all happen or constrain what can ultimately be done.

It's the systems architect's job to create this specification and do so with the customer so both parties know what's to be built. Detailed specifications make it easier to estimate how long the project should take and (more importantly) know when you're done. It's only by experience that consultants and developers learn how long a project will/should take. If the customer comes in halfway through the project (or a week before it's supposed to be done) and asks for something new, it's also easier to say, "Sure, but that's not in the specification, so it will take N more hours/days/weeks/months to make that change and it will cost X bucks more. Is that in the budget?" If the project specification is written as you do the project, it's not a specit's documentation. Yes, of course, this is an idealistic assessment of how things work, but if you're not prepared for this (very typical) eventuality, you won't be in a position to do anything but bend over and take it.

IMHO

If the project specification is written as you do the project, it's not a specit's documentation.

A specification is the end-result of the design phase, which (in my opinion) is far more important than the coding or implementation phases. Lack of an adequate design is the cause for many a project to fail, run over budget, or be unable to sustain the live production workload. Was it inadequate design and testing that caused the Boston tunnels to fall apart? I expect so. The project specification should be written before or as you build proof-of-concept routines that implement and test application components that have not been tried before. How many of you have seen production applications deployed from the prototype proof-of-concept code? Many of us are guilty of this blunder. Building a prototype is akin to testing a new type of siding or roof for a new home under actual weather conditions before committing your customer to something that might melt after the first hail storm. Two weeks after you go to production is not the time to discover that the chosen architecture does not have the needed performance for a scaled-up production application. Sure, as you gain experience, you'll know what parts and techniques you can depend on to perform when the load increases.

Tip

Be careful. Don't let your customer take the proof-of-concept application and take it into production thinking the prototype (held together with wax and chewing gum) is the real thing.


Choosing the Right Data Access Interface

This is not a book just about ADO.NET (although I'll spend a lot of words discussing it and showing you how to use it)it's a book about solving data access problems with Visual Studio and SQL Server. However, one of the problems I've seen of late is the tendency to use ADO.NET where it does not belong. ADO.NET (as DAO, RDO, ADOc, and the low-level data interfaces) are simple query and, some would say, update interfaces. None of these interfaces is particularly good at handling large rowsets or for moving data from one place to another. Using ADO.NET to move data from one system to another is like moving a ton of coal in a bicycle basket. ADO.NET is designed for, optimized for, and should be principally used for executing queries, managing client-side data, and posting minor updates to a database. I know there are many, many situations where you need to move data from one system to another. I spent a significant part of my time at Electronic Data Systems (EDS) using custom utilities to do just that.

SQL Server has always included tools for importing and exporting data. One of the oldest of these is the Bulk Copy utility (BCP). SQL Server 2000 added a new tool called "Data Transformation Services" (DTS). SQL Server 2005 replaced DTS with "SQL Server Integration Services" (SSIS). To make things even easier for data access developers, ADO.NET now supports a new SqlBulkCopy class to permit you to upload data from virtually any data source to SQL Server. Why go to the trouble of using BCP? Well, in one word: performancestartlingly fast performance. What might take days using ADO action commands can be done in minutes to hours. Because BCP, SSIS, and the SqlBulkCopy method use proprietary high-speed interfaces that bypass much of SQL Server's overhead, they can work far faster than using UPDATE or INSERT SQL commands.

When you need to make bulk changes to the database but these changes require complex logic (but isn't driven by user input), some developers have (unwisely) chosen to fetch many thousands of rows from the server, process them on the client, and send back the updated rows to the server. This approach is typical of developers who are not familiar with SQL Server stored procedures or other server-side executables. Consider that you can transport bulk data from virtually any trusted or untrusted source into a "temporary" SQL Server table. Once on the server, it's simply a matter of coding a stored procedure to munge the new data and merge it into the permanent (trusted) tables using whatever business rules or referential integrity constraints make sense. There is rarely a way to make the case to have a client perform bulk operations using ADO.NET queries and updates.

Choosing the Right Data Access Provider

As I'll discuss when I get to the later chapters, ADO.NET is simply a portal to a variety of data access "providers." Each provider exposes standard classes to implement custom native protocols (using managed code) or OSFA protocols like ODBC or OLE DB. These OSFA protocols are simply bridge interfaces to custom ODBC (managed code) drivers or COM-based (unmanaged) OLE DB providers written for specific backends. As a rule of thumb, it's best to use the most efficient protocolwith ADO.NET that means using a managed .NET data provider like SqlClient for SQL Servernot an OSFA interface. As a result, I won't be discussing the other providers very much (if at all), but I will make it abundantly clear how to use the SQL Server SqlClient provider in later chapters.

Given that developers sometimes wish to create a single application that needs to connect to a variety of backend data engines, using SqlClient can be problematic. In this case, it might be better to use a "generic" data interface that permits you to subclass the low-level data access calls and not have to specify a specific DBMS interface class. ADO.NET exposes new "factory" classes that make this process easier. That said, I'm not convinced that it's possible to write an efficient, competitive generic application. While it's possible to create classes that are backend agnostic, you may access only those features on the backend server that can be accessed on all other targeted backends. This approach means you might not be able to use stored procedures, as there are no two DBMS engines that implement them the same way. You won't be able to use XML datatypes, CLR functions, user-defined types, ANSI-SQL extensions, and other features that are not implemented universally. If you stick to SQL Server implementations like Workgroup, Standard, Enterprise, and Express, you'll have a better chance of creating a single application that can work on a wider variety of platforms.

What About SQL Server Everywhere Edition?

Just as the final chapter of this book was turned in for editing, Microsoft re-re-released a CTP of its free "SQL Server CE" database engine as "SQL Server Everywhere" (SQLEv). This CTP release was preceded by the SQL Server Mobile 3.0 edition, which was targeted (by license) for use on Windows Smartphones, Pocket PC, PDAs, and Windows Tablet PC systems. Microsoft previewed this announcement over a year ago at an author's summit, where I gave it my support. I could see any number of interesting and important ways to leverage this tiny DBMS. Over the last year or so, I worked with the Product/Program managers at Microsoft to refine this "new and improved" version of SQL CE. I've decided to publish an EBook on SQL Server Everywhere[7], so this book would not be delayed any further. The first release of SQLEv is expected to be in November, 2006, so it should be available on www.microsoft.com/SQL/Everywhere.

[7] See the support site www.hitchikerguides.net for details on the EBooks "Hitchhiker's Guide to SQL Server Everywhere" and "Hitchhiker's Guide to SQL Server EverywhereReplication."

When compared to other versions of SQL Server, SQLEv is different. Different should not be equated to bad or taken in a negative lightin this case, it means better performance, cheaper development costs and more secure databases. Permit me to illuminate a few of these similarities and differences.

Implementation

  • SSEv is implemented as a set of DLLsnot a service like SQL Server. This is an advantage in its typical use scenarios as it does not have to be started or stopped to release its resources. This makes Xcopy or embedded deployment easy and safe. I discuss several deployment strategies in the Ebookone of which does not require administrator rights.

  • SSEv database files can use the default file extension of .sdf but do not need to have any particular file extension. Unlike SQL Server, a SSEv database cannot span more than one file. In addition, SSEv databases are backward-compatible with SQL Mobile so that data files can be passed in both directions between applications.

  • SSEv has the same database size limitation as SQL Server Express Edition (4 GB), but there are plans to remove this SSEv restriction to permit far larger databases.

  • SSEv can be included in a bootstrap deployment package, so it can be used in a web- (or intranet-) based click-once scenarioSQL Server deployment is still a lot more complex. SSEv can also be deployed using a Microsoft Installer (MSI) to permit automatic updates to the DLLs through Windows Update.

  • SSEv does not use named instances like SQL Server. Each application that uses SSEv simply references the DLLs to access a private (albeit sometimes shared) database file. This also means you don't have to worry about whether or not SSEv is "started" or pausedit's always available.

  • SSEv databases (unlike SQL Server) don't have a "name" like "Pubs" or "Biblio"they are referenced through their data path such as "C:\Databases\Accounting.sdf". However, the file can be moved at any time (unlike SQL Server)at least as long as the file is not open. Erasing or moving a SSEv database is bad unless your application can automatically relocate the .sdf file.

  • SSEv cannot link to other databases like SQL Server (and JET) but it can replicate to SQL Server which can link to other servers or instances of SQL Server.

  • SSEv SQL uses a simpler way to refer to objectsyou don't have to (can't) specify an "owner," "schema," or alternate database but you can change databases in SSEv SQL or codeas long as they aren't password-protected. I expect that issue to be fixed.

  • SSEv databases can be attached to email or simply copied as a file to any target platformsomething that's very difficult for SQL Server databases. This makes installation and deployment a snapthe database and all of its DLLs (about 1.7MB uncompressed) can actually fit on a floppy or in the limited memory on a Windows Smart Phone or PDA.

  • SSEv databases don't use a .log file to manage their atomic operations (like SQL Server)logging is done, but via shadow pages managed by SSEv. While a .log file might appear, it's used as a temporary workspace when the database is compacted.

Security

  • SSEv is as secure as any DBMS engine provided by Microsoftif not more so. It has a far smaller surface area of attack which makes it far easier to secure.

  • SSEv databases can be password protected, encrypted, and further protected by the Windows file system encryption and rights schemes.

  • SSEv runs in process with the application, so it's not vulnerable to many of the attacks made against stand-alone or service-based DBMS systems (like SQL Server) that might be exposed to the network.

  • SSEv does not run as a service and it does not listen on network ports.

  • SSEv has a code-free data file architecture, which means it does not have the ability to run externally supplied code.

  • SSEv does not have the ability to run imbedded secondary SQL commands inserted by hackers which means it's not as prone to SQL injection attacks.

Sharing and Replication

  • SSEv supports a sophisticated replication interface that works with BCP/DTS/SSIS (to some limited extent), multiple subscriptions, configurable compression, column-level tracking and synchronization progress status events. Replication synchronization can be executed synchronously or asynchronously. I'll show you how in my EBook focused on SSEv Replication.

  • SSEv does not include the new (for ADO.NET 2.0) SqlBulkCopy method, so you'll have to programmatically configure client-side replication routines to upload data.

  • SSEv databases can be replication Subscribers to SQL Server versions that support replication Publicationthat does not include SQL Server Express. This means if you expect to replicate an SSEv database, you're going to need SQL Server Workplace or a better edition. However, in the next release of Visual Studio (Orcas), it's expected that SSEv databases can sync with other SSEv databasesone on a device and another on the desktop.

  • SSEv can support up to 256 connections but with the same caveatsa practical limit of simultaneous connected operations is closer to 70. However I don't expect you'll need more than a couple of connections for most applications.

  • SSEv's sharing mechanism is designed to share the .sdf database file with applications on the same systemit's not designed for multi-user or network file-sharing configurations like JET. It does support multiple connections from different processesbut all on the same system.

  • SSEv data files can reside on a file share on another computer. It doesn't support multiple machines sharing the same data file as this would fall into the multi-user scenario. However, users who keep their documents on another server for backup reasons can use SSEv data files. This isn't the case with Express where files must physically live on the server machine.

  • SSEv databases can be shared among applications (including SQL Server Management Studio and Visual Studio) on the same system but must be closed to perform repair, manual compression, and validation. There are also sharing issues when trying to replicate the database that I discuss later. Unfortunately, there is no way to tell if other "users" have the database open other than trying to perform one of these maintenance operations. There is no way to force the other users to close their connectionsas there is with SQL Server.

  • SSEv database connections do not support connection pooling (they don't need to), and the database engine can't be (and does not have to be) paused, stopped, or started (like SQL Server).

Programming and Maintenance

  • SSEv uses unique techniques to manage data that makes it especially interesting for a number of lightweight data storage needs, but these techniques are different from those used by SQL Server. These innovations include the SqlCeResultSet class that can implement a scrollable, updatable, bindable, in-memory cursor, as well as the TableDirect CommandType. When coupled with a named index, a TableDirect query can permit high-speed index-ordered access to a selected table.

  • SSEv's functionality is exposed via the SqlCE namespace. This is similar in some respects to ADO.NET (1.0) so it's missing functionality like SqlBulkCopy, connection GetSchema, and a few other classes. This keeps SSEv light but also might cause you to search for alternative techniques to solve some problems easily addressed by ADO.NET 2.0.

  • SSEv exposes a SqlCeDataAdapter, but Visual Studio does not expose a wizard to populate it. However, the Visual Studio TableAdapter wizards work against SSEv databases, so you might not find a need to use the SqlCeDataAdapter (at least not directly).

  • SSEv's data provider "implements" a subset of SqlClient functionality so it sometimes behaves differently. Its system classes can be used to populate System.Data DataTable and DataSet classes with a few restrictions (and surprises) that I'll illuminate later. I'll caution you now that knowing SqlClient programming can be a disadvantage when working with SSEv. While the objects, properties, methods, and events might look similar, there are many differences in how these objects are implemented. These differences can play a significant role in how you design your application and how it interacts with SSEv.

  • SSEv exposes a number of maintenance methods like Compact, Verify, Repair, and CreateDatabase that imply that the database needs periodic compaction, verification, and repair. Actually, after version 3.1 database compaction is done automatically but you still might find a need to repair and verify your databaseand no, this is not the same kind of compaction that your cat gets when it eats too many mice. You'll also find that SSEv databases are far less fragile than file-share databases like JET.

  • SSEv's compact and repair feature can be done only when there are no live connections. If you have a corrupted DB, you can use repair to remove the corrupted pages. The Compact method is more like a disk defragmenter. It will move all the free pages to the end and shrink the file at the end.

  • SSEv 3.1 and later support auto shrink (automatic compression), true ACID support, and row-level locking. The database is automatically compressed when the last connection is closed.

The SQL Server Everywhere SQL Engine

  • SSEv does not support triggers, stored procedures, SQL Views, temp tables, or familiar SQL Server metadata tables (like sysobjects) and most system views. In addition, its SQL language is a subset if TSQLcontaining just the commands you need to manage a client-side database. In some ways, this is an advantagebecause SSEv does not support any form of programmability, it's not possible to embed malware into the database. The Microsoft team is considering some sort of trigger mechanism (events that fire on data change) but this won't appear until a later version.

  • SSEv SQL supports a subset of SQL commands. This means statements like TRUNCATE <table> and SELECT Name FROM SYSOBEJECTS won't work. There are system views such as INFORMATION_SCHEMA.TABLES that expose lists of database objects like columns, indexes, tables, and RI constraints. Even some of the more common operations like a simple SELECT INTO to create a new table are not supported. This also means some of the features added to TSQL recently are not implemented in SSEv's SQLincluding TOP, ROWCOUNT, and others.

  • SSEv supports the ability to "seek" based on a named index valuea feature inherited from ISAM database engines. Just remember, it's not supported on SQL Server so don't get spoiled. It also exposes the TableDirect CommandType to support the ability to seek, find, and return rows based on a selected index.

  • SSEv SQL supports DDL commands like CREATE DATABASE and CREATE INDEX, but the syntax is only similar to TSQL and less sophisticated. SQL Server Management Studio also does not script SSEv objects to SQL as it does for SQL Server objects. While you can create a database with SQL, you can't alter anything but the tables. This means you have to use code to compact the database if you need to change the password or encryption attributes.

  • SSEv supports Identity and rowguid columns, as well as column defaults, and CONSTRAINTs but not RULEs, or specific column collation settings. However, there are no GUI tools provided to help create PK/FK relationshipsthis can only be accomplished by using an SQL script to create (or alter) the database tables. You'll also find there's no mechanism to disable identity value generation during data import routines.

  • SSEv supports full atomicity, consistency, isolation, and durability (ACID) operations just like SQL Server. This is essential for target platforms that experience power outages (most PDAs and cell phones run on batteries) and communications link failures (most platforms depend on wireless connections that are prone to connectivity disruptions).

  • SSEv queries do not support multiple statements in a batch (like SQL Server). This means no "multiple resultset" scenarios are possibleeven though the NextResult method is exposed. Don't be fooled by the fact that this seems to work in SQL Server Management Studioit's playing "helpful" tricks behind the scenes.

  • SSEv returns different errors than those returned by SQL Server. This means that everything from the simplest of syntax errors to the most complex synchronization errors are exposed differently. I discuss error handling later in my EBook.

  • SSEv also does not support [bracketing] reserved wordsyou have to use "double" quotes to delineate reserved words in SSEv SQL. They tell me this might be fixed.

  • SSEv supports a subset of the SQL Server datatypes and none of the CLR UDT datatypes. This means that VarChar, VarChar(max), Char, Text, XML, Timestamp, and several other familiar and heavily used SQL types are not supported. You will find NVarChar, Nchar, and NText are supported, and the Microsoft team tells me that Timestamp might be part of a subsequent release. By reducing the number of datatypes and simplifying the SQL language, SSEv can be smaller, faster, and less complex to program.

SQL Server Tools and Visual Studio Support

  • SSEv databases and queries can be managed with Visual Studio's Server Explorer and SQL Server Management Studio. This permits developers to tune queries with a graphical query plan, setup subscriptions, and perform DTS/SSIS operations. There are some disconnects with SQL Server Management Studio such as the inability to directly update or generate SQL scripts from Tables. SQL Server Management Studio also does not support the ability to "syntax check" SQL commands as it does for SQL Server queries nor does it support the ability to script database objects like tables. I've been told that some of these issues are being addressed in future versions.

  • SSEv supports indexes (and a query optimizer), referential integrity, and constraintsall of the basics for a relational datastore. It also supports dynamic schema alteration which many of its competitors do not.

  • SSEv databases can be completely encrypted and password-protectedSQL Server databases cannot. While you can encrypt selected columns in a SQL Server database, its security scheme depends on protecting the SQL Server's host system physical securitySSEv does not.

  • SSEv cannot be monitored with the SQL Profiler or the new Trace tools. This means you're on your own when it comes to debugging what's being sent to SSEv.

  • SSEv databases can also be used as a data source for Reporting Services or any other product that can accept data from a DataReader.

  • SSEv is not intended to run on IIS. Yes, there is a documented unlocking mechanism if you really need to enable this scenariobut I'm not providing it.

  • SSEv's documentation is not up to the same standards as SQL Server. There are many disconnects in the indexes; the topics are all-too-often incomplete and sometimes just wrong. Add to that the three-generations of help topics and web-based content that talks about SQL CE, SQL Mobile, and now SSEv, you can appreciate how hard it is to find help online. Now that SSEv is targeted to more application platforms, it should get more funding and attention from the documentation teams.

  • And finally, SSEv is not a V1.0 product. It's been around for many years as SQL CE and more recently SQL Mobile. Over the years, it's been tuned, debugged, and refined so it's ready for more platforms and more applications.

Don't get me wrong. I like SQL Server Everywhere. It addresses a host of problems that developers experience with JET (and SQL Server) and provides a fast, light, and fairly flexible DBMS platform. It's a welcome edition to the database application developer's toolset. I provide a lot more detail on where SSEv makes sense, how it works, how to implement and deploy it, and how to maintain it in the field. If you're interested in more information on SSEv, visit the support site for information on a couple of EBooks that discuss SSEv in far more detail.

Using the SQL Server Native Client

Microsoft has also added a new data access layer beneath ADOc when you need to connect to SQL Serverthe SQL Server Native Client (SNC). This new COM-based interface is designed to replace all ODBC and OLE DB interfaces to SQL Server, but without use of the MDAC stack and its associated issues. This means you can take your existing ADOc applications and recode the ConnectionString property to bypass MDAC completely, which should alleviate many DLL Hell issues. In addition, SNC is designed to expose all of the new SQL Server 2005 features, so your Visual Basic 6.0, middle-tier, or ASP applications will be able to use multiple active resultsets (MARS), XML datatypes, and more. I'll show you how to use SNC when I get to the ADO.NET chapter.


Choosing the Right Database Management System

I expect you're reading this book because you or your customers have already planned to use SQL Server or have an existing system that uses SQL Server and you need to be able to access it. I also expect my readers to be those migrating from Oracle, DB2, JET, or other databases and need some guidance on how to best transmogrify their applications and DBMSs to SQL Server. I am well aware that SQL Server might not be the best choice for every application. However, since it comes in so many "sizes," it certainly applies to the vast majority of application architectures being used today. There are many factors to consider when choosing a database management system, as I discuss here.

What Does SQL Server Cost?

Except in some back-street shops in Hong Kong, Microsoft SQL Server Workgroup Edition or better is not cheap. The licensing fees are computed on a per-system or per-CPU basis, where each additional individual user is required to purchase a Client Access License (CAL). See Microsoft for details, but you'll find that the cheapest (nonfree) version (SQL Workgroup edition) with five CALs runs about $739. That makes it pretty expensive to bundle SQL Server Workgroup edition with your small office accounting application unless you plan to charge quite a bit for it or expect the customer to have the right version already installed. You also need to consider licensing costs for the operating system that hosts SQL Server. The Workgroup edition can be hosted on Windows XP (SP2) or greater, Windows Small Business Server 2003 (SP1) or greater, and Windows 2000 (SP4) or greater. MSDE and SQL Server Express editions can run on the same mix of OS versions, but that still leaves out your customers still running Windows 95, 98, or ME, which are no longer supported. Other costs associated with SQL Server include deployment, setup, data loading, training, and administration, which vary quite a bit from implementation to implementation.

Thankfully, it's also possible to build small systems around the "free" MSDE (SQL Server 2000) or SQL Server Express editions. Sure, you'll want to develop against the Developer ($50) or Workgroup editions but deploy with the free MSDE or SQL Express editions. These editions are performance and scalability-limited, which make them inappropriate for systems that call for higher scalability. However, they're ideal for single-user applications that have to deploy with a DBMS, smaller office applications, low-volume websites, or even fewer-user multi-user systems. I have heard from a number of (very) large companies that have implemented many tens of thousands of MSDE-based systems in their organizations as point-of-sale systems, small accounting and registration systems, and other fairly sophisticated implementations. I further discuss these free editions and the special restrictions they impose in Chapter 2, "How Does SQL Server Work?"

What About SQL Server Security?

Any system you build needs to address security issues from the very earliest stages of the designbefore you choose a strategy or DBMS engine that does not have sufficient power to protect your customer's data. Security comes in many forms, degrees, and flavors. Of course, a secure system prevents data loss and prevents unauthorized data access, and, ideally, a secure system can be physically stolen without the data being compromised. While you might not be able to afford the amount of extra overhead needed to implement a totally secure system, you're obligated to provide at least the basics. Fortunately, a healthy percentage of the security in a SQL Server database management system is implemented automatically by the SQL Serverunless you disable it. SQL Server has a wealth of security features that make sure the data is correctly and safely stored. JET and many other "home" database engines have far fewer of these features, and those features that are supported are often done so at a very shallow depth. Recommended security functionality includes (but is not limited to):

  • Integration with Windows domain authentication or SQL Server login/password credential validation. This permits you to define specific users or roles that can be granted or denied specific rights to the database(s), tables, and columns, as well as procedures and views.

  • Encrypted (SSL) data transmission over the channel to and from the server and the ability to encrypt any or all data stored in the system using custom stored procedures (which can be written in T-SQL or custom CLR code). In SQL Server, this is implemented with Transport Layer Security (TLS).

  • Proprietary datatypes implemented in CLR code. These custom datatypes can prevent "bad," malformed, or out-of-bounds data from being stored in the database in the first place. These checks can also be implemented with rules and custom constraints.

  • Data transaction logging ensures that data that's supposed to be saved to the database is actually committed to the disk to maintain data and referential integrityeven if the server goes down before the data is physically written to the database. This mean the server can automatically "repair" the DBMS by rolling back uncommitted transactions and rolling forward any committed transactions that were not (yet) written to the server. It also means that transactions can include two or more servers with the DBMS engine providing "two-phase" commit as required.

  • More sophisticated systems support log shipping and data mirroring to move critical data off the server, so it can be more safely protected and more quickly restored and brought back online when things go awry.

  • Data validation subsystems that permit you (as a developer) to specify defaults and rules for the data so that regardless of the source of the data, it meets specific criteria such as range, datatype, and value. These can be implemented in SQL Server by Constraints, Rules, Triggers, or other server-side executableseven those written in a CLR language.

  • Stored procedures (triggers) that can be executed each time a row is inserted, changed, or deleted, to ensure that the data change does not violate business rules or referential constraintsthat you define.

  • Referential integrity and index subsystems that ensure that duplicate data rows are not inserted by accident and that parent-child relationships are maintained to further protect referential integrity and to improve query performance.

As you compare your current DBMS with SQL Server or as you try to decide whether or not you need these features, consider that these same features are available in the "free" MSDE and SQL Express versions of SQL Server. Without them, your data (and your job) is far from secure.

IMHO

ADO (or any data access query interface) is not usually the source of performance problemsbut it can be, especially when compared to low-level protocols.

What About Performance?

I constantly hear about "performance" and making applications run faster. Everyone seems to be concerned about performance differences between ADO, ADO.NET, DAO, or LSMFT[8]? Naturally, when choosing one data access interface or DBMS engine over another, developers want to be sure that the application they're designing is more responsive, supports more users, and is more competitive than the application they are replacing or competing with. Before getting into ADO.NET's performance, I want to touch on another point. ADO.NET (or any data access query interface) is not usually the source of application performance problemsfrankly, it's rarely the problem. Far more frequently, I've found that expensive (a.k.a. "dumb") queries, improperly configured tables, incorrect indexes, and other factors like bone-headed designs and simple ignorance play a far larger role in overall application performance than improperly coded ADO.NET routines. The problem is not how quickly your program (ADO.NET) asks the question, but how long it takes the network and SQL Server to return the answer. Sure, ADO.NET can be slow if you return too many rows for it to handlebut whose fault is that? Remember, it's designed to be a query interface, not a bulk operations interface. So can you blame poor performance on the database engine? Those that do are on pretty shaky ground. SQL Server as implemented today wins TCO/performance comparisons in some of the most complex and demanding DBMS installations in the worldespecially when price of the hardware is considered as a factor. Sure, Microsoft sometimes uses a bit of smoke and mirrors when building marketing demos, but I've interviewed enough developers (and their managers) to know that SQL Server more than meets their needs for a deadly serious high-performance platform.

[8] LSMFT: One of the first acronyms. It means "Lucky Strike Means Fine Tobacco."

When you start looking for a performance bottleneck, look first to your database design and configuration and next to the queries you're submitting for execution. For example, INSERT performance can be hampered by too many indexes, but SELECT performance can be hampered by not enough indexes. Overall performance can also be crippled by sheer volumetoo many rows being returned, sorted, filtered, related, or updated at once. By asking the server to return an entire table, state, region, or country, you can transform a relational database into a file server and your client application into a leopard slug[9]. I'll talk about appropriate high-performance query syntax in Chapter 2 and in subsequent chapters throughout the book.

[9] www.fcps.k12.va.us/StratfordLandingES/Ecology/mpages/leopard_slug.htm

IMHO

It's often not how quickly you ask the question, but how long it takes the DBMS engine to return the answer.

Performance Metrics

Benchmarking performance is one of the most complex and exacting data processing sciencesand one of the most misunderstood and misapplied. In order to make accurate (and fair) comparisons, you must set up a level playing field (if one exists) so that the hardware platforms don't favor one system over another. But what if they do? Perhaps, the fact that system X runs best on platform Y but other systems do not should play a role in the evaluation. For example, if you discover that SQL Server runs better on hyperthreaded or dual-core processors, but the alternative does not and the target platforms are HT systems, then SQL Server would make more sense in that situation. I recommend using a typical production client or server system as a testbed so your tests can better reflect reality.

Without boring you with a chapter on how to set up and interpret the results of a benchmark, consider that one mistake that many folks make is testing with insufficient data, using the wrong kind of data, or using atypical queries. The Pubs, Northwind, and AdventureWorks database are not nearly large enough to really judge anythingunless you're planning to put them and the data they contain into production as is. When testing performance, it's hard to beat dealing with real datanot made-up data rows that don't reflect the actual variety and distribution of your customer's data. To give a simple example, when SQL Server executes a query against a table, it can choose which index is most appropriate. Suppose your sample Customer table consists of three million rows of a duplicated "Mrs. Smith" record indexed on last name and date of birth, but only one of those Mrs. Smith records has different data in a non-indexed field. Your test query's WHERE clause matches "Mrs. Smith" and the non-indexed field. At best, the performance you'll see reflects a table scanno matter how hard you try. If you compare that result against real data that has only 300 "Mrs. Smith" records interspersed in 3 million other rows, you'll see an efficient index scan over 300 records rather than 3 million. The point here is that when weighing different performance approaches, be careful that you are weighing the same thing.

IMHO

Just because a duck floats, it does not make a woman a witch because she doesn't.

Comparing ADO and SQL Server vs. DAO and JET Performance

Consider that ADO is and always has been a query interface. Until Version 2.0, ADO.NET did not have any way to move data in bulkat least, not efficiently. Unfortunately, developers tend to use bulk data transfers to measure ADO's performance. For example, they compare how fast ADO can retrieve 10,000 or a million rows against some other DBMS like Paradox, Oracle, or JET. Using this technique to benchmark is like comparing the speed of a bicycle against a Porsche 911 in Redmond rush-hour traffic (with the cops around). ADO, ADO.NET, and SQL Server have been compared countless times with DAO and JETand JET often wins. When both DAO and ADO.NET are used to access JET, DAO often wins. Despite this apples-and-can-opener comparison, these tests sometimes come out in JET's favor because DAO is a "native" interface to JET. In contrast, ADO.NET uses a one-size-fits-all OLE DB interface designed to access any number of different types and classes of data sources. All too often, developers migrating from DAO (and JET) bemoan ADO.NET's (and, thus, SQL Server's) performance. Sure, DAO and JET are often faster than ADO.NET and SQL Server for some operations (especially when talking about bulk data transfers or simple index-seek operations), just as a flat-file is far faster to read than setting up and executing a JET query and fetching the data. ADO.NET (and especially ADOc) moves a lot more metadata over the wire that the lower-level interfaces don't bother withthey assume the application knows the schema and structure of the data being sent and received. Yes, ADO.NET is better at this, as it separates out the metadata from the "real" data. In some cases, this metadata can make your application safer and smarter, and make it easier to update the data when the time comes. But most of the time, it just gets in the way. You can minimize this overhead by choosing when to use the DataAdapter or TableAdapter Fill methods and when to simply use a DataReader to execute queries. But I'm getting ahead of myself. All through the book, I'll show the cost of each data access technique and how to best optimize your network, RAM, and CPU resources on each end of the wire.

What's Fast Enough?

As a developer, you should be concerned with achieving "acceptable" performance for the types of queries your application must execute and the ability to scale those operations to support the load demanded by the expected users per second. Consider that a server architecture (like SQL Server) actually gets faster as more users (doing roughly the same things) are added (up to a point)not slower (like JET). Why? It's because as users perform roughly the same operations on roughly the same data, more relevant data and procedures are fetched from RAM caches and do not have to be fetched from disk. In (stark) contrast, JET and other similar ISAM database management systems cannot "share" or reuse data pages fetched by other users, as they are distributed client-resident engines and do all of their physical I/O over the Net and lightly cache only what the individual client is accessing. They have no idea what other clients are doing beyond the lock status of one or more rows or tables. This means JET can support far fewer users and is governed by the speed and capacity of the Net and available CPU cycles on the client system.

Sure, if you're building a single-user application, SQL Server (even MSDE or SQL Server Express) can seem like overkillJET or another lightweight DBMS engine can be very appealing. However, how many times have you been asked to convert a simple single-user application to multi-user? I've heard of this happening quite often as a small-office application gains popularity and others want to share the data and functionality. I've learned that if an application is not designed to be multi-user from its inception, it's not likely to succeed as such after conversion from single-user.

But what about SQL Server Everywhere (SQLEv)? This new DBMS has the potential to replace many JET databases where a simple, fast, and light data store is called for. In contrast to JET, SQLEv is fully encryptable and can be an ideal local repository for client data. I discuss this in detail in my EBooks on SQLEv.

Optimizing Developer Performance

As you design your application, a more compelling scenario needs to be considered. It concerns developer performancethis is especially important for development and IT managers. Developer performance is measured by how many solutions can be designed, developed, tested, deployed, and supported in a given length of time with a given set of developers. It's not how many lines of code are developed in a given a length of timeas developers, we can easily "bulk up" our programs if we're being paid by the line. When you build any application, you and your team learn, tune, and leverage specific skills. The techniques you apply control the specific features of the DBMS that make your application perform well and integrate well with other applications. The exception handlers you write, the administrative functions you include, and many other aspects of the application are specifically tuned to accommodate that specific DBMSwhether it's JET, SQL Server, Oracle, or 3 x 5 index cards. This makes sense.

However, when your customer asks you to scale the application or simply start from scratch and create a new application that's clearly out of your current DBMS's reach, the skills and techniques you and your team have honed may no long apply. As a result, you as a developer are not nearly as useful to your customer or employeryour entire career might be placed in jeopardy. If, on the other hand, you learn to use ADO.NET and SQL Server, you can leverage skills, techniques, schemas, procedures, views, and a wealth of other resources for a project of almost any scale. Retooling the design for more users is not that great a stretch, given your new skillset.

A number of respected experts suggest the developers build intermediate "object" layers to which the development team codes[10]. This can eliminate the need for your team to learn ADO.NET (or the data access interface dejure), but it does require more training and discipline to learn how to design, implement, use, and support this custom interface. The Microsoft Data Access Application Block (DAAB) is a variation of this approach. When you decide to switch your backend to another DBMS engine, (in theory) you can do so by replacing the data object layer. Ah, yes, but this is also fraught with complexities and hidden dangers, as there always seems to be unintended side-effects when removing one component or layer of components and replacing it with the "equivalent."

[10] See Building Applications and Components with Visual Basic .NET, by Ted Pattison and Joe Hummel (Addison-Wesley Professional, 2003).

Another issue your developers are going to have to deal with is "builds," service packs, and the intermittent flow of "fixes," patches, and upgrades that pop out of Microsoft and the other vendors on which we all depend. Once your application is deployed, it must (somehow) continue to work even when many of the interfaces it uses are evolving. While .NET architecture alleviates some of these problems, I've still seen situations where upgrading the existing .NET framework breaks deployed applicationsas when a service pack is applied to the framework. I've also seen problems purportedly caused by the tighter (somewhat anal) security "improvements" in Windows XP SP2. For the most part, if you're reading this book, you're pretty smart and you should know how to debug and work through most of these upgrade issues. Unfortunately, your users aren't always that informed. Because of this, developers have to deal with these users when they're just doing what they're told to do and upgrading their systems. In an ideal world, an "upgrade" or patch should not break their applications. But life, love, politics, and software development are not that simple. It's our job to make sure that they and their applications don't failat least, to some extent.

What About Shared Code and Team Development?

When companies develop applications using SQL Server, they often do so in teams, as the projects they build are far too complex to be completed by a single (or even a married) person. Usually, there are too many interdependencies for a single developer to handle by themselves, so other developers help with design, writing specifications, setting up databases and data interfaces, creating user interfaces, writing documentation and test procedures, and doing everything else that goes into a serious projectincluding making tea. A critical aspect of these more complex designs is the management team that coordinates the activities of the development team. Without disciplined management, these projects invariably degrade into chaos. That's where Team System comes in. This new version of Visual Studio is designed to help teams architect, design, map, and build, as well as archive and protect complex applications under development.

In shops both large and small, developers usually have to work with a specialized, dedicated database administrator (DBA) who's responsible for data and referential integrity and system maintenance, as well as a full plate of other duties. Sure, in lots of shops, the developer is the DBA is the developer. This is where protected objects and shared code play a critical role in the success or failure of the project and in the security of the database. Based on my research, DBAs generally prohibit unrestricted access to base tables in a database but grant execute rights to specific stored procedures or views to query and alter data. As I explain in Chapter 5, "Managing Executables with the Server Explorer," these stored procedures are set up very much like COM objects with (hopefully documented) named interfaces, named parameters (input and output), pre-defined return values, and any number of rowsets, print messages, error messages, and event log entries. Each of these procedures leverages SQL Server's ability to pre-compile a query plan that matches a given set of input parameters to further optimize the procedures' operations. By using stored procedures, a development team can more effectively and efficiently manage a complex application development project.

IMHO

Building designs around stored procedures requires something some shops simply don't have: discipline.

However, building designs around server-side executables requires something that some shops simply don't have: discipline. Once applications are written using one or more stored procedures, functions, or user-defined types, it's essential that no one (not even the DBA) change the code without understanding (and testing) the impact on the applications that use (or will use) the server-side code. The same can be said for any DBMS objectstables, views, column, user-defined types, rules, or any database object. This is even more important when datatypes, stored procedures, functions, and aggregates are written in CLR code. If the DBA (or a developer with DBA rights) makes a change in the schema, it can easily impact (irritate, antagonize, madden) others who depend on a specific schema layout for their applications. Yes, it's far harder to develop complex applications in teamsespecially if the individual members of the team aren't communicating or are running open-loop with their own agenda. Does this mean holding never-ending meetings to discuss these changes? Sure, if that's what it takesbut hopefully not. Better-disciplined teams use a variety of techniques to communicate changes to other members or simply delegate all changes in the form of documented requests to a DBA. Sure, the vast majority of the sites out there don't have a full-time DBA watching over the systems and developers to make sure they don't do something dumb. What usually happens is that one (or more) developers take on the role of the DBA on an informal basis, or perhaps the task is relegated to an off-site support team. What's tough is when the role of "DBA" includes everyone in the organization from the development team to the janitorial service and everyone in between. This can happen pretty quickly when the SA password is posted on the cafeteria bulletin board or handed around to anyone needing access to the database.

Building Shareable and Reusable Components

Shared code issues are not new. This problem started when companies started trying to build products with replaceable partsthe idea was that one could build an appliance, car, or software program with components built to be used in one or more products. That is, the same part could be used in copies of the same product or in other similar products. For example, Ford could use the same updraft carburetor in its 1930 Model A Tudor or its Model A truck. If the part "wore out," it could be replaced with one just like it, and the product would be back on the road again in no time. This concept revolutionized industry and product manufacturing all over the world. Instead of using skilled craftsmen to build custom (and often complex) products one by one, industries turned to more efficient mass-production assembly lines. While this required more complex designs and parts manufacturing had to be more precise, it made the manufacturing process simply a matter of fitting the parts together on an assembly line.

This concept works fine when it comes to relatively simple components like interchangeable windshield wipers and exhaust mufflers or SCSI/IDE/SATA hard drives. But when the products improve and a new model is shipped, do the product designers include these older parts? Sometimes, but not alwaysthey often improve them (that's the whole idea behind "new and improved"). But do the newer parts work in older modelslike a USB 2.0 drive in a USB 1.0 port? Sometimes, but not always, not nearly as frequently, and usually with some degradation in features or performance.

Let's stick to software instead of how well replacement car parts work (or don't). Shared software components are a lot easier to build than a Ford four-cylinder short block. They don't have bearings that wear out (or, at least, none of my applications do), but they do have lots of parts and often many interfacesthe "connectors" used to hook up the components to other components or your application. When these shared or "common" components are upgraded, it's usually (but not always) done without changing their name or the existing input or output parameters, so they expose the same datatypes, names, and values as the older version (this is called the "signature")they "look" the same as the component they're replacing (at least to the older program). This way, the new component can be installed without having to change the existing applicationit simply asks for a component by name and the OS loads itat least, in theory. However, the inner workings of the shared components change even if the external name, signature, or interfaces don't. These changes are either "bug fixes," "enhancements," or simply side-effects of other changes. Good or bad, in any case, the component behaves differentlyhopefully in positive ways, but any change invariably has unintended side effects. Since developers using a component have to work with the existing bugs or behaviors, when the bug is fixed or the behavior changes, the mechanism can fail to work in the way it did before the change. This can break your application, make your documentation incorrect, or simply confuse the user.

Sharing the MDAC Stack

Consider what happened when Microsoft "fixed" ADO 2.1 and introduced 2.5 (you don't want to know what happened to Versions 2.2 through 2.4). In this case, the MSADO15.DLL used to implement all versions of COM-based ADO) was replaced by a new DLL (with the same name), but the functionality of the new DLL was so different that it broke applications from one end of the planet to another (except in some parts of Cleveland). The real problem is that developers often have no choice when it comes to updating their applicationsthe MDAC stack (which contains ADOc) is included in new versions of Windows, Microsoft Office, Visual Studio, and other tools and applications, and is quietly installed without developers (or users) really knowing what's going on behind the scenes. No, it's not possible to use a specific version of COM-based ADOyou must use the version that's installed on the target system. Yes, you can code your application to address a specific type library (typelib) for one version of the COM component or another, but you'll still be executing the same binary. Each version of the MDAC stack included typelibs for older versions. For example, ADO 2.7 includes typelibs for 2.1, 2.5, and 2.6. When you code your application, you point to a specific versionthe current DLL or an older version's typelib. This way, your application still sees the signature of the older version, even though it's (always) implemented by the most recent MSADO15.DLLthe new version "fakes" the interfaces exposed by the older version. Sure, your application could install the latest version at deployment time, but if a newer version was already installed, you couldn't (shouldn't) overlay ityour only option is to hope your application developed and tested with the older version still works with the new "improved" version of ADO. Sadly, it sometimes doesn't. This catch-22 is called "DLL Hell." If you've been working with COM-based ADO "classic" (ADOc) for any length of time, you're doubtless familiar with the issues.

The .NET Revolution

The .NET Framework was invented to eliminate (or at least minimize) these very serious DDL reuse and sharing problems. In this case, applications can build an executable and test, debug, and deploy it with a more-or-less guarantee that the code will continue to work even if newer versions of the .NET Framework are installed (again, at least in theory). Yes, this means that your target system might have 2, 3, or 12 versions of the Framework installed, and several might be in memory at once. However, the applications should be able to function independently (assuming there is enough RAM and other resources). Sadly, I have seen some evidence (in the fall of 2004) that SP1 for the 1.1 version of the Framework has broken some existing applications. Given what I've said about "replaceable" common code, I'm not really surprised.

Each deployed .NET application uses its own Bin directory to hold its unique executables. Yes, this area on disk might contain copies of components used in other applications, but updating one application does not affect the others that share the componenteach application gets their own copy. It also means that different versions of the same component can (and do) exist in memory at the same time. .NET applications can be built using COM components, so .NET permits you to either reference a shared (common) copy that can be replaced or upgraded, or a private copy that belongs exclusively to your application assembly. As I'll show you later, continuing to use COM components (like ADOc) in your .NET applications won't help performance, but you don't have to worry (as much) about DLL Hell.

What About Multi-User Issues?

When designing your application, you need to think about how many users are going to be accessing the data at any one time. It's relatively easy to write an application that simply shares data with lots of peopleas long as no more than one person tries to update the data at the same time. This is as easy as printing a memo and handing it out at a meeting. If you're building a single-user application, the only contention you have to face when working with the database is from your own application. Sure, it's possible (and not that uncommon) to lock yourself out of a database by not thinking about data flow and what's being done behind the scenes to supply requested data. When you execute a query, the database engine finds the data referenced in the SQL and starts to return it to you. Until the process of passing that data to the client is complete (rowset population), the server can hold share locks on one or more rows, pages, extents, or tables, or even the whole database, depending on the scope of the query. If you try to update the data you just (started to) read, you might find that you're blocked until the rowset population is complete. I talk about this again when I discuss the DataReader in Chapter 8 and the Fill method in Chapter 10, "Managing SqlCommand Objects."

Once you plan to support two or more users with your application (running multiple instances of the application), the rules of engagement change and your application's code becomes an order of magnitude more complex. In this case, you can't afford to simply read all of the rows from one or more database tables into memory, update as you please, and write back the changes to the database. That's because your changes won't be seen by other users (or even by other parts of your own application) until you post those changes to the database server and the data is requeried. If other users change the data while you're holding it in memory (and possibly changing it), your code has to figure out whose changes are to take precedence. When two or more applications change the same data, it's called a "collision," and it's your job to clean up the mess caused by these data collisions.

If you're using a "disconnected" data architecture as commonly used in ASP.NET (web-based), ADO.NET "disconnected" applications, or "load and go" applications, your changes might not be committed to the server for quite some time. In the meantime, while your user is deciding what to buy, sell, or change, someone else (or a thousand others) could be doing the same thingfetching some, all, or none of the same data and making changes. If everyone has rights to change any and all data, what happens when someone tries to update the data you fetched? ADO.NET does not have any mechanism to automatically decide what to do or prevent changes to the data still in the database[11]it's up to you to tell ADO.NET what to do if there is a collision (when an update is attempted and another change was made to the data since you last read it). I discuss techniques to avoid collisions and to deal with them once they occur in Chapter 12 "Managing Updates".

[11] Yes, it's possible to set up a "pessimistic" lock to prevent access to rows, pages, or tables, but that's not an option for most architectures. I'll talk about locking in Chapter 8 "Getting Started with ADO.NET" and Chapter 12 "Managing Updates".

ADO.NET 2.0 does have the ability to work with SqlDependency objects that can be tied to a SqlCommand. Once properly configured, your application can receive an event if selected data on the server is changed. This is great if you've populated a pick list from a list of valid widgets. If some operation on the database changes members of your dependant Command object's rowset, an event fires to notify you of the change. I plan to write another EBook on this subject.

What About Scalability?

When you build an application that's designed to support more than one user, you're well along the way toward building a scalable application. Scalability simply means the application design automatically (or gracefully) supports a growing number of users or operations in a given length of time. Not all systems need to be scalable beyond specific limits. Often there is no reason to go through the (rather expensive) steps of building in scalability if your application does not need to support more than a dozen to a few hundred users. Sure, virtually all applications can scaleat least, to some extent. It's how well they scale that makes the difference. A well-designed application should scale linearly. That is, the performance should remain about the same as more users are addednot increase exponentially. As more users are added, a scalable system should be able to accommodate additional load, just as a properly matched engine, transmission, and drive train can handle pulling that 3.5-ton boat and trailer behind your SUV (assuming you bought the towing package).

Highly scalable designs should lend themselves to scaling "out" as well as "up." That is, applications that can scale out must be able to support large numbers of users and are coded so as to spread the load over several servers (clustered servers or server farms). Are scalable applications faster? Frankly, not always. I've seen that the techniques used to make an application scale up and out usually add to system overhead, so they don't perform as well as single- or few-user applications. However, these techniques are often a prerequisite to the ability to scale. In other words, the customer interface is not particularly snappy (as in a single-user application), but the system can support tens to hundreds of thousands of users with "reasonable" response times.

In the past when discussing scalability and replication, people usually meant designing systems that used several servers. Now, with the advent of mobile devices capable of hosting SQL Server Everywhere (SQL Mobile), scalability through replication can involve many hundreds, if not thousands, of "clients" all trying to stay in sync. It's clear that future scalable designs need to consider mobile devices such as smart phones, PDAs, television set-top boxes, and home appliances. Sure, before long, many of us will be writing programs to keep your refrigerator in sync with your PDA, so you won't forget that you're out of low-fat milk when you go to the store.

What prevents an application from scaling? Generally, it's because it does not know how to "share." Consider that an application must run in an operating system that's used to support other applications. If the server is dedicated to SQL Server, all the betterall the application has to do is consider other SQL Server users and their demands for resources. If your application demands server-side resources such as server-side cursors and temporary tables, consider that, for each connection made to the server (one or more for each user), any demand on server-side resources is multiplied. Sure, SQL Server can support lots of activity on TempDb (where cursors, intermediate sorts, and temp tables are created), and it knows how to efficiently manage memory to cache the most frequently used data pages. But if you expect to create a scalable application, you'll learn to conserve these resources and offload some of the work to the client system. For example, you might be able to reduce your application's resource footprint by deferring a sort to the client application or not using a server-side cursor (which is tough to do in ADO.NET anyway) or not building large temporary tables. However, by efficiently using server-side state, you might also reduce the amount of work done by the server each time your application needs to execute a query or perform an update. The trick is to learn how to balance these resources to maximize efficiency.

For ASP applications, scalability (up and out) is essential. In this case, the application your customer executes by navigating to an .ASPX page is executed for a very short length of time, but a significant amount of time is spent rehydrating the application session state. That is, the overhead involved to reset the application variables to the values set when the application was last accessed by this user. This can include restoring a saved data state as well or re-running a query to refresh the data state. Restoring this state can be expensive, so it's essential that as little "state" as possible be saved.

What Are the Limits?

The most efficient application in the world has its limitationsjust try to log on to one of the web sites mentioned in a Super Bowl ad during the game. The best applications know how to deal with excess volume as gracefully as they deal with normal demand levels. It's very educational to see how some of these web sites fail when stressed by tens of thousands of bored fans. We've all seen folks build systems that demand too much of their hardware. Consider that SQL Server, Oracle, or DB2 are world-class database management systems, but they all have limitations. Even when given one or more high-performance hyper-threading processors to execute the code, high-performance hard drives to fetch and store the data, gigabytes of fast RAM to cache the data, high-speed LANs to return the results, and efficient code to execute, any DBMS still has a top-end transactions-per-second limit. When the code is less than optimal, the disks are slower or fragmented, the CPU is clogged up with unrelated overhead, you don't have much RAM, or the LAN is busy streaming videos, your overall system performance can be degraded. When you use a client computer to host SQL Server (as with MSDE or SQL Server Express installations), you're often faced with a system that's short on everything hardware-wise, not to mention the fact that the user might be running Doom in the background. It does not make much sense to test and tune your new application on a development system that can radically out-perform the target user's system. Users rarely like to hear, "Well, it works just fine on my system...." It's our responsibility as developers to write code that's efficient enough to run on the available iron. Blaming hardware is like a farmer blaming the plow for the stones in his field.

IMHO

It's our responsibility as developers to write code that's efficient enough to run on the available iron. Blaming hardware is like a farmer blaming the plow for the stones in his field.

Does this mean you need to plunk down $9,000 for a new system to run SQL Server to run a few dozen users? Hardly. Consider that in the late 1980s, Microsoft IT ran a 386-33 server to support thousands of campus client/server usersand they were running the OS/2 version of SQL Server. SQL Server is capable of handling a tremendous volume of connections and work. Over the years, it's been tuned and refined to work even more closely with the operating system, so it's one of the most (if not the most) efficient DBMS platforms on the market. This means to support SQL Server in a small to medium-size organization or a moderately active web site, you might not need anything more than a bare-bones cast-off system that's capable of running Windows NT, Windows 2000, or Windows Server 2003 (depending on your security demands). I like to use my second- or third-oldest systems to run SQL Server for my sites. I suggest the same for some of our customers who need to support a few dozen to a few hundred users. It often means the customer can use a $400 system to do the job. If the system gets overloaded as the company gets more successful, it's a snap to upgrade to faster hardware. Sure, it means that our applications have to be especially efficient and not squander the somewhat limited hardware resources. If you write sloppy code and ignore our suggestions, you might not get away with a cheap server system.

IMHO

If you write sloppy code and ignore these suggestions, you might not get away with a cheap server system.

What Can Get in the Way on the Client?

When applications and databases are deployed, it's important to know what other applications your client application (which might be running on an IIS server, a hand-held PC, or a Windows 98 system) and your SQL Server are competing with. I've seen very few companies that give their employees top-end Pentium systems with lots of RAM as their client workstations. While your customer might be an exception, more typically these systems are older PII or PIII systems with very little RAM and tiny (by today's standards) 5,400 RPM drivesand that's in Trump tower. In many companies, I've seen far less capable systems. Companies simply can't afford to constantly upgrade their systems to accommodate inefficient applications or more feature-rich operating systems. This means it's our responsibility to write applications that are efficient enough to work on the client systemsas is.

All things should be taken in balance, thoughPeter developed a database system for use in local government where the indigenous hardware was adequately specified for running a copy of Microsoft Excel and Word concurrently without too much thrashing. However, once he installed the application, things went south when the customer wanted to run Excel, Word, and his application at the same time. He knew that he would hit that problem, as he was aware of the customer's hardware configuration. Given the situation and politics, it was cheaper for him as a software supplier tofactor in the cost to upgrade the system's RAM as part of the software cost rather than trying to engineer our application to use (far) less memory.

Consider that your application has to share hardware resources like CPU time, RAM, screen real-estate, and disk space on your customer's system with other applications including RAM-hungry word processors, ancillary background tasks, anti-virus tools, SQL Server instances, or even games. Sometimes, upgrading the hardware as a "software upgrade/fix" satisfies the anal bean countersand your customer.

Unless they are dedicated application workstations, client/server Windows systems are typically burdened with other applications like Outlook, Word, Excel, a browser, and a multimedia player or a game or twoall running at the same time. Sure, your application can compete with these applications if it's given enough resources (or does not demand too many resources) to stay in memory. However, we've all seen the new pariah: ad-ware, spy-ware, and viruses that can suck CPU and other resources out of your client system like kudzu sucks the life out of a tree. If your shop is highly disciplined (and I've seen some), it does not permit the users to run anything on their work PCs that has not been cleared and validated by the company. When the applications running on a Windows system demand more memory than the system has available, Windows chooses one or more blocks to swap to disk. This can dramatically slow down the system. A faster hard disk can help, but not as much as more memory. If SQL Server is loaded on the workstation, it knows how to release memory on demand, but you can configure it to use less memory in the first place.

IMHO

There's nothing that slows down SQL Server like a busy print or report serverexcept perhaps a Castle Wolfenstein screen saver.

What Can Get in the Way on the Server?

With server-based executables like IIS, ASP, and SQL Server itself, you need to be just as vigilant to avoid running jobs or extra services that compete with the system. Sure, this means it can be a lot more efficient to run SQL Server, Reporting Services, and IIS on separate dedicated serversif your customer can afford it (few can). Consider that SQL Server Express edition with Advanced Services can host Reporting Services (on local data) along with full Text search, so price might not be a factor. It also makes sense to put your print spoolers and exchange, ISA, and file servers on separate systems. When you don't need expensive licenses, split off server functionality to separate servers. Consider that it might be your own applications slowing down the server. As we've discussed in this chapter and I'll repeat throughout the book, a single poorly written application can bring a system to its knees. Sure, your application might not be at fault, but it could be blocking (or consuming) resources needed by another application that's chewing up CPU and disk resources like my granddaughter chomps on blackberries.

There's nothing that slows down SQL Server like a busy print server or Report Serverexcept for a screen saver. These both use CPU-intensive rendering engines that can exhaust a system's resources in no time. I got a complaint from a user that his client applications were running very slowly, but only sometimes. When he went to the server to see if the performance monitor was showing any bottlenecks, the server came back to life and began to run normally. A few minutes later (always about 10 minutes), the server slowed down again. I suggested that they turn off the 3D "Castle Wolfenstein" screen saver that was sucking every bit of processing power from the server.

What About Maintenance and Administration?

Every database application system needs to consider DBMS maintenance and administration as part of its responsibilities. We can't assume the user will periodically back up the database and know how to restore it when things don't go as planned (as they always do). We also can't assume the user will know how to set up new users or assign appropriate security rights. Sure, this responsibility can be delegated to a DBA if they are aware of these requirements. But if you're building an application that does not have the luxury of even a part-time DBA, you need to build-in the appropriate maintenance tasks.

When working with JET databases, these tasks are of critical importance, as JET requires periodic compression and repair. Since JET does not reclaim space lost when rows are updated or deleted, the database can grow out of space and your customer will be helpless to address the problem. Backup with JET is also problematic. Since JET won't let the backup (or compression) utilities run while any user has the database file (or lock file) open, you'll have to make sure all users are logged off before continuing. This can be problematic if Fred has left his system running and goes on an extended vacation.

Managing Maintenance Tasks

SQL Server is a lot easier to maintain, but the basic administrative tasks still need to be done. As I'll discuss in Chapter 2, SQL Server logs each change to the database to the transaction log. The log grows automatically (as does the database), but there's no need to compress SQL Server databases as space is reclaimed as rows are deleted and updated. However, you'll want to build routines to automatically back up and flush the transaction logs and periodically back up the database when building any application that has no dedicated DBA. These maintenance tasks can be done online, as SQL Server does not require that you log off users while these tasks run. All of the maintenance tasks can be done via T-SQL, but they can also be done via System Management Objects (SMO) in SQL Server 2005, which exposes a programmatic interface to .NET managed objects, properties, and methods to back up and restore the server. SMO is a completely new object model built from the ground up that is intended to replace SQL DMOalthough DMO is included in SQL Server 2005 for backward-compatibility only. Your application might also need to perform some (or a great deal) of login account setup and management, depending on the security approach you use. Note that expensive user rights management schemes might not be necessary when implementing SQL Express.

IMHO

While periodic backups are vitally important to your customer, they pale in comparison to the importance of "restores."

The Importance of Restoring Data

While periodic transaction log and database backups are vitally important to your customer, they pale in comparison to the importance of "restores." The most comprehensive backup in the world is useless if your application has not figured out the complexities of a system, database, and transaction log restore. All too often, I've seen customers lose hours, days, and their very existence because they were unable to restore their databases. I recommend that you start with a virgin computer and document every aspect of the process needed to bring that system back to a working state. This documentation must be stored both onsite, where it can be found quickly, and offsite (with the data), where it can be found by those who survive a more serious disaster. Don't discount the difficulty of repeating the process of resurrecting a system. Be sure to consider what you'll have to do if the Internet and online sources for software updates are not available. I like to build "ghost" images of the base system to expedite the recovery process, but it's really beyond the scope of this book to get into the nitty-gritty details of this process. Just be sure you and your application are ready. Your customer will thank you for it.

Keep in mind that backing up or even imaging a system drive won't do you any good if the replacement system has a different motherboard. I've had considerable difficulty when replacing dead (or dying) motherboards with an upgraded version. It seems that the installation routines that make low-level hardware configurations are executed only during initial Windows setup, and those cast-in-stone configurations prevent the OS from running on different hardware. This means if your disaster plan calls for system replacement, it had better be identical systemsperhaps ones that are many years old, if you expect a simple image restore to work.

Dealing with Backup and Deployment Issues

It's essential that you take into account the foibles of the hardware used to store your user's (and your) data. Hard disks (like any mechanical device) have a limited life. While some drives last longer than others, typically you should expect 3 to 5 years for a $100 drive. Electronic memory that retains its state post power-down (flash and other technologies) is also subject to failure. Power supplies, fans, every moving part, and every exposed port in the system are all subject to failure due to mechanical wear, dust infiltration, or voltage variations. Every type of storage is subject to failure of some kindeven clay tablets or concrete tombstones. This means you need to back up your data and be ready for the hour when the components fail. More important, you need to be able to restore your systemquickly, quietly, and efficiently, without your customer noticing. Frankly, the latter is harder. It's easy to execute the back up routines in any DBMS, and SQL Server's are especially easy to use. TSQL backups can even be done from your application. A new feature in SQL Server 2005 (SP1) implements fail-over clustering and database mirroring. I wish I had more bandwidth to explore that with you, but again, this is targeted for another EBook.

Restoring your system to its previous statenow that's the problem. When working with SQL Server (as I discuss in the next chapter), you should take a periodic log dump to save the transaction logs to removable media. The base database backup plus the logs equals a restored database. If you're running more than one database on a server, you'll have N database backups and one set of logsone log backup for each week/day/hour you back up (and dump) the logs. Having backed up the database and dumped the logs, you still can't restore if you don't have a base system to restore to. In my experience, when it comes time to restore, it's more likely that the entire system is poochednot just a single database. This means you might need to build a new hardware system up from a system image (using Ghost or the equivalent), or by handone step at a time. The manual process can take quite some time (days, in some cases) of configuring and loading, updating, posting web updates, installing service packs, and an endless list of configuration settings. That's why I encourage you to make a system image of the completed system once it's installed and ready to accept the database restore. Once you're ready, you can restore the base database backup and then restore the log files one at a time. This should return your database state to the point of the last transaction log dump/backup. That's why, for serious production systems, the DBA arranges to take relatively frequent transaction log backups and transports them off-site immediately to protect the database in case of a catastrophic problem.

IMHO

Restoring a database can be an order of magnitude more difficult than backing it up.

Dealing with Security Issues

Once you code your own data access routines, you'll need to set up your own security mechanisms if the data is even slightly sensitive. Sure, if the program is simply inventorying the contents of the user's refrigerator, you (and the user) might not care if someone broke into the database to find out that they are low on cottage cheese. You might be a bit more security conscious if you thought your mom was hacking into your refrigerator system to check on your stock of cold beer. However, because your code implements custom data I/O routines, you can easily incorporate sophisticated encryption techniques in the data you store. This will make it very tough to break into the data and uncover sensitive informationeven with full access to the hardware. Security mechanisms include ways to gate access to the server and the client application itself. They can be as simple as setting up logins in the Windows domain or creating users, groups, and specific permissions on database objects. While I discuss the litany of security issues throughout the book, I want to make one point clear: Do not develop or deploy an application that connects to the server with the SA account and password. If you ignore this advice, you'll not only compromise your data security, but you'll also mask a number of issues that will crop up if you even decide to tighten security later. Of course, there is an exception to every rule. When you deploy a SQL Server Express database with a User Instance, it can be accessed as SA as the user is granted those rights. I discuss this in detail in Chapter 9, "Getting Connected."

Security is not something you tack on to an application or a server like a final coat of paint on that Mustang you're restoring. Security systems that truly protect your data and your customer's data (and your job) are sewn into the fabric of your design from its initial conception to its deployment and implementation. These systems can leverage the Windows domain to validate users, but this is not a panacea. I'll show how it's possible to hijack the user's credentials to execute malicious code or SQL operations. This means that any program that solely depends on "trusted" (SSPI/Windows domain) authentication can be used to execute a query that erases the database or, worse, exposes its data to those who should not have rights to view or change it. No, I'm not going to create a Hacker's Guide to SQL Server, but I know that if you're familiar with the issues, you'll take appropriate steps to minimize the exposed surface area.

The way I see it, there are at least three security personas:

  • The SA or administrator, who has super-rights to the database, schema, infrastructure, and data

  • The developer, who needs rights to the schema but not necessarily the data

  • The user, who needs to be able to view and change data, but just that data to which they are entitled

To implement a secure system, you (or your DBA) need to limit unrestricted access to the data and the schema, as well as the security systems themselves. The first line of defense is the server itself. This means that you (above all) need to protect the SQL System Administrator (SA) account. Sure, this means you'll probably have to find a dedicated DBA and ensure that this person does not release the SA password to anyone. In some cases, you won't even know the SA password. With the default configuration (which disables mixed-mode SQL Server security authentication), any Windows domain administrator (any Windows login that's a member of the Administrators group) is automatically granted SA rights. This means that there might be any number of "SA"s in your organizationeach a potential security leak.

Each object in the database and the database itself needs to be protected. This means every database, table, view, procedure, and everything in between, including data, schema, and the permissions assigned to protect them.

To make this process more difficult than it has to be, Visual Studio 2005 does not include any of the tools you'll need to set up or manage user accounts or permissions. I have been asking for these for at least four versions of Visual Studio, to no avail. This means you'll still have to use the new SQL Server 2005 Management Studio to create new users, remove existing users, set up groups and roles, and assign permissions to these users. Keep in mind that while Windows logins imply that a "person" is logging on to the server, I think you should set up "roles" instead. That is, instead of assigning permissions to "Fred", you would assign permissions to "AdminClerk1". This way, when Fred is promoted, his replacement (Sally) won't have to log in with Fred's credentials. Actually, when I create applications, most of them don't require the user to provide credentialsthe application provides these behind the scenes. In this case, it's important to carefully gate access to the front-end application or web page.

Okay, so if all of the objects in the database are protected, how is your application going to extract data? Clearly, you're going to have to open some controlled pathways to the data. The most frequently used approach is to write stored procedures that fetch and update the data needed by your applications and grant limited access to those procedures. This means that logins (other than the SA) can access the data only through these carefully scripted procedures. In any case, applications (even administrative tools and the Visual Studio Server Explorer) won't be able to gain access to the data except through these stored procedures.

This approach is fine for end-user executables, but what about your development team? In this case, you're going to have to work out an approach that exposes the schema but not (necessarily) the data. I often like to use a parallel database that contains data that's distributed n the database, like the live data, but does not contain sensitive information. It can't just be a million identical rows or even randomly generated data, as the query optimizations you develop won't work against the live data. By using a parallel database, you don't risk compromising the live (secure) dataeven though it's a bit more trouble to set up. Developers need to be able to see the schema but not necessarily see (or modify) the live data. They might need to create new tables, views, rules, triggers, and procedures, but these tasks can be restricted to the DBAand they are in many larger sites.

Understanding Development Costs

As development managers, I'm always looking at the cost to develop and support applications, as well as how quickly they work and how quickly they meet customer needs (and get them out of our hair so we can devote our time to the next project or a new customer). I've found that efficient coding yields more satisfied customers more quickly because the application stability and its ability to solve the customer's problems is higher. This often means that I don't universally encourage use of home-grown data access strategiesespecially if there is even a remote chance that I'll have to upgrade the application to use shared data anytime in the future. You'll need to balance this issue as well.

Many shops seem to prefer to roll their own code instead of using third-party controls or components (including DBMS engines), even though outside solutions might handle 8099% of what the project requires. This is called the "NIH" (Not Invented Here) syndrome, which has cost our industry billions as companies reinvent the wheel for their internal projects. And it's not just components from third (outside) parties that are shunnedsometimes these are components written in their own company but by another department. I can't count the number of times I've heard that an outside vendor's product (while good) could not possibly be as good as "something I build in-house from scratch." It turns out that this is rarely the casebut the revelation occurs only after many months or years of lost resources that have been flushed down the drain along with a number of people's jobs.

IMHO

The "NIH" syndrome has cost our industry billions as companies reinvent the wheel.

Managing Resources

Every application needs hardware resources like RAM, CPU time, disk space, I/O channel time, and network bandwidth. The best applications manage these resources like precious commodities so that there is enough to go around and to improve performance. In the olden days, I worked with mainframe computers with 256K RAM and a bit 40MB hard drive. CPU time was measured by the second, and each CPU second had to be accounted for, as it was extremely expensive. These systems were capable of handling hundreds to thousands of users using architectures not unlike those in use today in IIS and ASP.NET applications.

Sure, these hardware resources are cheaper today than ever, and the price is still dropping. However, this does not mean you can afford to waste them. A good developer knows when to trade off using a shared resource with a local resource. For example, should you re-execute a query to resort a rowset on the server or simply resort it in memory at the cost of local CPU time and memory?

The network is another generally misunderstood and squandered resource. Most of us run Ethernet networks that can tolerate quite a bit of traffic, but when they reach capacity, they tend to collapse quite quickly. Our applications that connect to and through corporate networks have to compete for bandwidth with applications that are streaming videos, browsing the web, running NetMeeting or terminal server sessions, and playing a shared game of Age of Empires. If you add to that the IM and email applications that constantly poll and pump for new messages, you get a pretty good idea why your download is taking so long to run. If the corporate backbone supports only 10MB or a slower Wireless network, you can count on applications timing out from time to time (especially around the holidays, when lots of people are browsing Froogle for their spouse's new toy).

RAM is another precious commodity that applications have to share with Windows and all of the running applications, as well as the printer and other device drivers chewing up memoryeven though you haven't printed, scanned, copied, or faxed in a week[12]. Add to that the dozens of services and extra little "helper" executables, and you no longer wonder why it takes so long to boot. Of course, if you spend any time at all on the web, your system will soon be overrun with a dozen adware, spyware, and remoraware applications reporting on your every click to those web sites in China and chewing up memory and CPU time. It's important to make sure your application knows how to tolerate a low-RAM situation and how to free resources when asked. Nope, you won't get an event that says "How about releasing some RAM?" unless you're really cleverthose details are handled behind the scenes for you by .NET (at least, to some extent). The best idea is to consider how much RAM your query is going to consume before you execute it and (even more so) after you create a data structure that has to be persisted in memory for any time at all.

[12] My development system had (at one point in time) seven HP printer DLLs loaded that consume over 32MB of RAMin idle mode. I've since switched to a Canon printer that has no memory-resident footprint that I can detect.

In Chapter 9, I'll drone on and on about connectionsthe mechanism to connect your application's data access routines to the backend DBMS. While this is a somewhat complex subject, you'll discover that connections are another precious resourceexpensive to construct and limited in number by license (in some cases). I'll talk about strategies that open connections just in time and close them immediately as the query finishes, and other strategies that open connections and leave them open. Both of these approaches make sense in the right context. I expect you'll write quite a bit of code to manage your connections and recover from exceptions triggered when the connection is closed unexpectedly or there are no connections available (because you used them up).


Understanding Basic Data Access Architectures

This section discusses several different application architectures, based on how they access data. No, not all applications need a DBMS engine (as I discussed earlier), but almost every application you design or build accesses data. This could be as simple as saving configuration settings or user preferences to a local file or as complex as extracting data from a web-based XML Web Service or a complex relational database. Each of the architectures discussed here has advantages and disadvantages. Each requires a different skill level, resource demands, and interdependencies. It's these interdependencies that can make applications especially complex. When your application starts depending on other platforms, people, and shared resources, it needs to be programmed to deal with a new set of realities. In our experience, applications most often fail because they don't adapt when their environment changeswhen business situations changes, when companies merge, when systems go down, when business gets bad, when business gets good. In other words, when stuff happens. For example, if your application depends on a specific database that becomes unavailable or simply changes in some way, your application had better be prepared to deal with this reality. No, it's unrealistic to expect your application to automatically morph itself to the new data structure, but it does make sense to use a development scheme that can more easily adapt to changing schemasespecially during the development process. Yes, object-oriented programming and strong type checking are a good start, but the question remains: What does your application do when it finds that the data structure it expects has changed, is missing, or is temporarily unavailable? What should your application do when the Net connection disappears? Perhaps, your code expects an integer but is sent a float or a long. While many development shops can't afford to write applications that deal with every possible contingency, these caveats should keep you aware of the issues and encourage you to take all the precautions you can to help recover from the application crashes that inevitably occur.

At this point, you should understand many of the issues that all database software architects facewhether they're designing an application for a team of developers or simply building a tool for themselves. The following section briefly describes each of these architectures and their principle attributes and limitations.

IMHO

The simplest application to build and maintain has the fewest number of interdependencies.

Multi-Tiered Designs

No book on application architectures would be worth much if it did not at least mention "tiered" application architecture design principles. This is not a new conceptit's been around for many decades. However, over the years, how architects view tiered architectures has evolved. We build tiered applications for many reasons. There are "good" reasons and "bad" reasons. A common "bad" reason is to complicate your application by breaking it into tiers because someone said it was the best thing to do. There are several good reasons, including separation of work, separation of logic, and database/data source independence. When you break up any application, you must create "interfaces" between the layers. In the COM days, one would create an object and expose properties and methods on that object that other tiers could consume. In the Service Oriented Architecture (SOA) world, one could create a stand-alone "service" (like an XML Web Service) that's accessed via the Simple Object Access Protocol (SOAP) protocol, which Visual Studio can help generate. Another approach is to simply move business logic into SQL Server implemented as stored procedures, rules, and triggersI've used that approach for decades. Regardless of how you expose the objects, properties (attributes), or methods of your tiers, publishing and documenting these interfaces is of critical importance. If these interfaces change (as they often do during development), the "object" consumers must be informed of this change so they can adapt, recode, recompile, and redeploy their applications.

Basically, the idea behind 2-tier, 3-tier, or "n-tiered" applications is to:

  • Separate the logical parts of an application to permit different groups to work on the application. For example business logic, user interface, and data access logic can be separated into separate tiers. This way, the "artsy" folks can work independently on the user interface and the "data" folks can work on the data tier logic.

  • Permit one or more tiers to be genericized, to permit it to be replaced at a later time. For example, one might create a UI to interface to an English-speaking user and another for a Spanish-speaking user. Another example would be to create a data access module that would permit switching between SQL Server and Oracle backend database engines.

  • Separate the parts of an application so they can be executed on separate processors or systems or remotely. Typically, I use this approach to improve performance and scalability by having each system execute just those tasks it is tuned to execute. For example, moving business rules to SQL Server stored procedures is a way to move logic into the "data" tier.

You can break these tiers down any way you want to, but generally they are broken down into the "presentation" (or UI) layer, the "business intelligence" (or BI) layer, and the "data" layer, as shown in Figure 1.1.

Figure 1.1. Three-tiered architecture.

[View full size image]

Each of the architectures I discuss in this chapter can be constructed using a multi-tiered approach. Sure, some architectures and applications and development teams lend themselves better to segmented designs than others. Do multi-tiered designs make your applications execute faster? Usually notthis approach usually adds more overhead as the logic flow moves from tier to tier over the interface "bridges" between tiers. Does this approach make development easier? Well, yes and no. If you choose this approach, it means the interfaces defined between the layers must be defined early in the development process and rigidly adhered toespecially if different teams (possibly different companies) are expected to develop and implement the separate tiers independently. In other words, the teams have to be disciplined to work together. In complex projects, this approach can make the overall project easier (sometimes possible) to codeit might be the only way to address the problems.

Creating Database-Agnostic Applications

Oftimes, I hear from folks who have tried to create a multi-tiered application designed to use either SQL Server, JET/Access, Oracle, or 3 x 5 cards as the data source. They hope to be able to "simply" replace the data tier and leave the BI and UI layers intact. Not many of these succeed. It turns out that there is more to a layer than just the interfaces. While it's possible to implement each of these layers with carefully constructed classes with "standard" interfaces (properties, methods, events, and arguments), what goes on behind the scenes is also very important. Sure, it's possible to create a generic OSFA data interface that simply constructs "appropriate" SQL to return a rowset or update a data table. However, most DBMS systems do many things differentlyit's their competitive edge. If you discount the 3 x 5 card system (which has a distinct advantage in situations where the power fails), JET, SQL Server, and Oracle all implement (so-called ANSI) SQL differently. Okay, so the data tier can accommodate those differences as well as the differences in the .NET data provider, the connection string, security, and the data types each supports (or doesn't). But consider that JET, SQL Server, and Oracle all implement stored procedures very differently. Their administrative functions are also radically different, so backup, restore, compression, repair, transaction log management, index maintenance, and a dozen more issues make each backend very different. Sure, creating database-agnostic applications can be done. It's just going to be more expensive than writing dedicated applications that take advantage of the special features and sidestep the shortcomings of the backend serverespecially if you consider that very few of the desirable backends are static: Most are evolving. With the exception of JET (I hope), all of these platforms are adding new features, new paradigms, and new ways to manage data more efficiently. A database-agnostic application will be hard-pressed to incorporate these new features as they are added.

Building Stand-Alone Applications

The simplest application to build and maintain has the fewest number of interdependencies. A healthy portion of the computer applications in use today are designed to be used by only one person at a timewhat I categorize as a stand-alone application. These applications usually save data to a local file and have very few dependencies to confuse the logic. I expect there are a thousand stand-alone applications that save their data locally for every application that requires access to a DBMS of some kind. Consider that Excel is the most popular "database" in use todayeven exceeding Access or other "real" DBMS applications. Excel applications depend on one or more spreadsheet "workbook" files to maintain the data state. However, there are even more applications that simply save the application data state or user preferences to a local file or the Registry. While stand-alone applications might collect information from the user and perhaps save and retrieve data, they don't always need a sophisticated DBMS to manage their data. Stand-alone "disconnected" applications include calculators, word processors, spreadsheets, accounting applications, games, thermostats, clock radios, and many others. Stand-alone applications also include custom applications written for and by your company to deal with special dedicated, often complex operations.

How many of you have looked over the counter at the screen used by hotel clerks, car dealers, travel agents, and other salespeople? I have. Clearly, one of the most widely used paradigms in use world-wide is not a Windows forms or ASP applicationit's CICS. CICS (Customer Information Control System), is the transaction serverbased system that (predominantly) runs on IBM mainframes and uses a 3270-style vector-graphic (line-drawing) or character-based screen. I got my start in data processing working with these systems. Sure, some of these applications are hosted on Windows using 3270 emulators, but Visual Studio is nowhere to be found. I expect that many of these applications were first written in the 1970s and 1980s. Why are they still in use today? Because they work. They're reliable, fast, and well understood. Why am I mentioning this approach here? Because when it comes time to choose an application architecture, you should consider the experience these systems can bring to bear. You also are likely to have to compete with them or interface to the data they collect and query.

While you can consider that process control applications (designed to run elevators, robots, complex machines, security systems, and the six-disc CD changer in your car) are typically "single-user" stand-alone applications, I'm going to skip over this type of application for now. Perhaps I can write about process control application development in one of my EBooksit's a fascinating subject.

Another platform for stand-alone applications is personal digital assistants (PDAs). Again, this is a rapidly expanding marketwhat I call a "target-rich" platform where there are lots of potential customers and more to come over the years. I bought over $300 worth of software for my PDA (and Smart Phones) in the last couple of yearsjust don't tell my wife. No, they weren't all games. The problem with this space is lack of support by the manufacturers. I've had to buy a new cell phone each time a new version of Windows came out for the platformMotorola did not want to (or couldn't) upgrade. The techniques, limitations, and tools used to develop these applications is also fodder for another EBookand beyond the scope of this book.

Sure, a stand-alone application usually needs to retrieve, manipulate, and save data. The question is, how is this done? Can these applications justify use of even a simple DBMS system like JET or SQL Server Everywhere[13], or can they get away with building a custom data management system on their own? Well, the answer is yesthey can do either. For decades, I've designed and written a variety of applications that don't require anything but a simple file I/O data retrieval system. Nowadays, you can save data to a file using your own custom data structures in a .NET stream, so you don't need a DBMS to persist (save) and retrieve your data. To make this even more powerful, ADO.NET (as I'll show you later in Chapter 8) permits you to build stand-alone DataTable objects and save them to a stream in an XML format. This makes "rehydration" (the process of rebuilding the data structures from a file or other external source) very simple.

[13] Be sure to check out my EBook, Hitchhiker's Guide to SQL Server Everywhere. See the support site www.hitchhikerguides.net for more information.

IMHO

Keep in mind that XML file storage is not particularly efficientsure, it's flexible and feature-rich, but not efficient.

Keep in mind that XML file storage is not particularly efficient. Even a small block of data can expand to several times its binary size when translated and encoded to Unicode (16 bits/character) and formatted as an XML data structure. However, with systems as fast as they are today and disk space relatively cheap, this should not present a big problem for a reasonably small number of data records. I say "records" because I'm not describing a relational database, but a flat-file data storeit's not even an index sequential access method (ISAM) database like JET, as the data structure does not include any indexes to speed data retrieval. Sure, if you use the XML Document Object Model and its associated methods, you can search XML data structures and retrieve selected data on demand.

As the number of data rows you plan to store in one or more local files increases, consider that ADO.NET supports quite a few methods and properties on its data structures (the DataSet, DataTable, and DataView) that make it easy to find, sort, sift, and filter the data once it's in memory. This means you might not need to graduate to a DBMS for some time, as you can build, save, and reload a complex DataSet from a number of separate files. However, this approach means that you won't be able to do cross-table "JOIN" queries on DataSet objects as you could with a relational DBMS engine, but you will be able to define your own inter-table relationships. Once the DataSet is created and populated, you can establish ADO.NET-managed parent-child relationships on its DataTables and use all of the relational features afforded to this in-memory DataSet "database," including binding and parent-child navigation and consistency validation.

IMHO

Another advantage to the stand-alone data file approach is the lack of a DBMS engine.

Another advantage to the stand-alone data file approach is the lack of a DBMS engine. Because this type of application does not require installation of a separate DBMS engine (and its inherent compatibility, DLL Hell, and resource issues), your application should be easier to deploy and more stable once installedno matter how many times MDAC is updated. If you look closely, you'll also discover that Visual Studio supports the ability to create DataSets (even strongly typed DataSets) based on "objects" that you define as classes. In other words, you'll be able to use many of the drag-and-drop code generators even though you aren't connected to a DBMS. I'll talk about this again in Chapter 6.

Deciding How Much Data to Store

So what's a "reasonable" number of records to persist in a flat-file data store? Well, when the user starts to complain about the time it takes to get the application going (as you rehydrate your DataSet-based "database"), you had better start thinking about a different, more efficient strategy. Unfortunately, once you've build a flat-file application and the entire support infrastructure around this approach, it's harder to "upgrade" to a DBMS than if you had started with a data engine in the first place. One performance alternative might be to switch to a binary file format instead of an XML structuretrading disk I/O volume for additional saving and rehydration code. However, this approach requires quite a bit more knowledge of binary code structures and more sophisticated persistence techniques.

If you think your application might need a DBMS engine at some time in the near term, it's probably best to start out using one instead of trying to redesign six months after you deploy the new application. Because the basic architecture of a DBMS front-end application is different than an application where your own code carries much of the data access burden, you'll often find its far more expensive to switch than start out using a DBMS like MSDE or SQL Server Express. In the same way, if you think your single-user application might be required to access shared data, it's essential that you start from an architectural base that best supports this data access requirement.

Dealing with Hybrid Architectures

Sure, a number of applications are not "pure"they often have to support stand-alone modes and keep working while the DBMS is unavailable, busted, or simply offline. For example, for insurance salesmen (the ones who are not geckos) or agents who work out of the office, it's not uncommon to use a local laptop-based data store that's synchronized with the central DBMS once a day or so. In this case, it's best to build your application as a shared-data application from the beginning and include routines that fall back on a local DBMS engine like SQL Express or simply log changes to be posted later to an XML flat file as I have been discussing. When it comes time to synchronize the data, you can let SQL Server do it for you using replication, or write custom routines that rehydrate a DataSet and post the changes to the DBMS using the Update method.

Figure 1.2. Client/server architecture.

Managing Client/Server Connections

The connection is a direct channel to SQL Server from the client. It carries the Tabular Data Stream (TDS) packets to and from the SQL Server query engine. These packets are used to transport TSQL strings, which can contain a SQL SELECT, other T-SQL commands, or utility invocations (such as database maintenance [DBCC] or bulk copy [BCP] commands). Since SQL Server supports multiple T-SQL statements in a single string, you can group several T-SQL commands together (with or without intervening semicolons). Each returns its own resultset, as I discuss in Chapter 2.

In a client/server application the "client" application is generally expected to open a network "connection" to the central server and leave it open for the life of the application. A client/server application (actually, any application) can open as many connections as needed. Yes, to better leverage the number of available connections and economize on server-side resources, applications tend to close unneeded connections during idle times and reconnect as neededas when it comes time to update the data. This means the application usually maintains state locally so that it can resume from where it left off without a lot of extra work and round-trips back to the server. The server connection also persists information about the "state" of the client and manages a number of scoped resources that can be seen or manipulated only by that client connection (and no other). These resources include temporary tables (#Temp tables) and intermediate sort results in TempDb, server-side cursors, and local connection-scoped variables such as @@IDENTITY.

In the 2.0 Framework, ADO.NET exposes a new feature called Multiple Active Result Sets (MARS) that permits (for some reason) multiple use of a connection by more than one part of the application. For example, you might need to populate a local rowset with the results of a query while at the same time executing an UPDATE on another set of rows. With MARS enabled (it's off by default), you can use the same connection for both operations. Frankly, I'm of mixed mind about this "feature." First, MARS adds overhead to connection and TDS management behind the scenes. It also breaks down once you attempt to open more than 10 connections. Next, it's not that expensive to open one or more additional connectionsat least, not to SQL Server. While some other platforms make you pay dearly for more connections (like Oracle and JET), SQL Server connections are relatively cheap to open.

When a connection is "closed," all server-side resources associated with the connection are releasedat least, they are if the connection is actually closed. Because of the way that ADO.NET and the .NET data provider pool connections behind the scenes, the connection-pooling mechanism does not actually close the connection immediately. This means resources remain allocated until the connection is actually closed, times out (after 48 minutes), or is reused. However, connection pooling is not an important factor in client/server applications, so you might want to disable this functionality.

The client application is responsible for connection management. As long as the connection is open, the server periodically polls the client to ensure that the client is still breathing. This means if your client is very busy playing a game or goes to sleep (goes into suspend mode), the connection might be dropped at the server end. Any number of things can break a connection, so your client application should implement fairly robust exception handlers to deal with these contingencies. If the connection is lost, you'll need to be prepared to reconnect and recover server state, as the server will drop all server-side state that belongs to your connection.

Each connection is expected to provide some flavor of authentication credentials that permit the server to grant access to the DBMS resources your application needs (and no more). These can be a login ID and password or a request to let the Windows domain help manage client validation. Based on these credentials, SQL Server grants, prohibits, or simply hides access to specific DBMS objectsincluding the server and database itself. I discuss connection management in more depth in Chapter 9.

Client/Server Data Entry Paradigms

Generally, a client/server application exposes forms to the user used to capture parameters to fetch relevant data and display the results of the queries. The application is also designed to capture user-generated data. If the user is expected to enter data from the keyboard, the client/server application has a much heavier responsibilityto pre-validate the data. This is an essential step for several reasons. Data validation includes ensuring that the data entered meets datatype constraints (integer range checks, float/integer, numeric tests, string-length checks, and precision and scale checks) and business rule checks (allowable range, minimum, maximum, and inter-field relationships, to name a few). An extremely important reason to check user input has to do with SQL injection attacks, a technique used by hackers to gain access to or damage your DBMS and the data it protects. Later in Chapter 10, I discuss the details of this security problem, but suffice it to say here, it's very important that you not permit users to provide strings that are blindly inserted or concatenated to SQL statements to be executed on the server. Be sure to learn how to use an ADO.NET Command object before executing any queries, as this can usually defeat this (very common) type of hacker attack.

IMHO

Be sure to learn how to use an ADO.NET Command object before executing any queries that include user input, as this can usually defeat SQL injection attacks.

Sometimes (often), business rule validations can be done only by accessing lookup tables. In some cases, you might be able to download validation values from lookup tables and force the user to select from one of these pre-validated values. This is a unique advantage of the client/server architecture, as your application can easily afford to download this data. The brief time it takes to populate client-side validation tables is miniscule in comparison to the time saved by preventing server-side validation failures. Another approach to this task might be to save lookup table data in a local data store like SQLEv.

And here's the first of many catch-22 situations. How can client/server applications validate user input without hard-coding business rules? There are several suggested solutions to this quandarywhich of these are viable is another question:

  • Use a mechanism to fetch the business rules from the server when the application is first started. Some time back, I wrote an article[14] describing how to do this using SQL Server extended properties. Basically, it works like this: Each object in a SQL Server database can have one or more extended properties assigned. These properties can be given any name or value and can be assigned to the database itself, tables, columns, views, or any other object. For example, you could assign "minimum", "maximum", "default", "mask", or any other property value to an individual column. With a bit of logic, you could build a state machine to validate user input. When it comes time to change these values, it's a simple matter of changing the values on the serverthe next time the application starts, it can fetch the new values. Of course, you can devise a way to signal to the applications that new parameters are available so that this client application update process can be done automatically. One technique introduced in the 2.0 Framework is client notifications, where the client application can be notified by event when a table changes. However, this approach would be problematic if events were lostas they appear to be when the engine is under heavy load. If you limit the notifications to cache sync you should have more luck.

    [14] http://www.sqlmag.com/Articles/Index.cfm?ArticleID=20886

  • Store the business rules in a configuration file that's managed and deployed separately from the base application. This might be a suitable way to use XML to persist and rehydrate your client-side business rules. A better solution would be to store the business rules in a SQLEv database, as illustrated in my EBook.

  • Since SQL Server now permits you to store XML in the database as a native column type, you might consider creating a business rules table to help manage client-side rules and other business logic. Again, you could use client notification to signal a change in these values, so the clients could fetch them as soon as something changesas long as you don't over-stress the engine.

Managing Client/Server Data Validation

It's important (and easier) to pre-validate data as it's entered instead of trying to deal with one or more server-side update failures. In many cases, the user is working with an input document or is talking to the "real" source of the dataanother person. If the user has to wait until the entire form is complete before validation errors are detected, the whole data-entry process can be slowed down and made far more difficult. If you submit an entire form at once to validate, the exceptions that are returned are often tough to trap and manage. You might also find that these exceptions do not include all of the errors, as the server-side update logic might stop at the first exception and not discover remaining exceptions until you try the update process again. This might mean you'll have to attempt the update operation two, three, or more times before it actually succeeds or the user gives up in frustration, throws the monitor through the nearest window, and crams the keyboard down the nearest toilet[15]. One approach to deal with this problem might be to attempt "pseudo updates." That is, instead of asking the server to actually perform a complex update, you can submit a query that simply tests the validity of the data just entered. That way, when you actually execute the complete form update, you can be relatively certain that the update will succeed. These pseudo updates can be (should be) done behind the scenes without the data entry clerk even knowing that they are taking placeassuming you can master asynchronous queries. This feature is reenabled in the 2.0 Framework, as I'll discuss in Chapter 11, "Executing SqlCommand Objects."

[15] I think it's interesting that people take their frustrations out on the monitor and keyboard, which are rarely as guilty as the computer itself.

Limiting User Choices

I'm of the opinion that most errors are caused by humanseither those programming the computers or the users entering data. No, not all problems are caused by politicians or "evil doers," but many are caused by users given access to a keyboard. As I see it, there is (approximately) a 1 in 103 chance that a user will hit the correct key when asked to type in a response. I still hear of people wanting to know the location of the "any" key ("press any key to continue"). To address this issue, I encourage (and use) "pick lists" of valid options when building data access forms. This way, users must choose from a list of valid thingslike states, sizes, options, and part numbers. Sure, when building client/server applications, I have the option of preloading or reloading these pick lists based on what the user is doing now or did yesterday, or what I expect him or her to do. For example, if the user chooses Bellevue, Washington, as the city and state, I already know the postal codes for that city. Frankly, it would be easier for us if they pick the zip/postal code first so I would immediately know that they should be entering the matching cityI actually do that for them. All the user has to do is verify that I chose the right city and state. The United States Postal Service (no, that's not an obscure Windows service) makes machine-readable postal codes available for this type of validation query[16].

[16] See www.usps.com/ncsc/formoreinformation.

Pick lists can also be populated based on (ever-changing) business rules using the techniques I just discussed. This way, when dealing with a special customer (one that deserves a special discount or one to which you would rather not extend any additional credit), your pick lists can adapt automatically. This means you don't have to train your clerks (as much) to deal with these special cases.

IMHO

The real test for a data-entry form is the ability to pass the "cat" test. That is, if a cat walks (or takes a nap) on your keyboard and your application doesn't fail, you've done a good job. Note that there is very little that can protect your system from other cat ejecta and bodily emissions.

Managing Client/Server Data Retrieval

Unlike some other architectures, the fact that the server is just a brief query away makes it far easier to populate browse lists and perform more complex searches. A browse list is a (relatively) long list of data rows requested by the user. A client/server application can easily cache these lists indefinitely and manage their content locallyfetching and refreshing the list as needed. For example, if a user wants to see local used car lots, they might initiate a query that returns quite a few rows. Let's assume the user does not live in Copperas Cove, Texas (where there are only 14), but in a city with several hundred dealers anxious to take your money. I think it makes sense to return no more than a few hundred rows from a query before you ask the user to narrow the search. This is not really a problem with used-car dealers, as there aren't that manyeven in bigger cities. But if the user wants to locate all of the nationwide retail outlets that sell Barbie dolls, they would have to deal with over 10,000 sources (not counting the black market outlets and guys hanging out in alleys near elementary schools selling cheap knock-offs to unsuspecting kids). In this case, the volume of data is far too much for a user to "browse"even if you used very tiny fonts. To address this issue, I offer the following words of advice:

  • Fetch just the rows the user needs now and no more. If the query is too broad, suggest (or anticipate) ways to narrow the search. You especially need to avoid sending rows the user does not need or throws away. When I get to the ADO sections of the book, you'll find out that it's not all that simple to discard rows once you ask for them in a query.

  • Limit returned rowsets to no more than a few hundred rows. If you design around smaller query result rowsets, your application will scale better, perform better, and yield happier customers.

  • If the user wants to visually scan a lot of rows, fetch the first couple of screens first (20100 rows) and display them immediately. In the background, fetch the next couple of screens so when the user clicks "down" to see the next set of rows, you're ready. The code behind this technique involves use of TOP expressions and a WHERE clause that helps segment the fetched rowsets into smaller groups.

Tip

Remember that SQL Server is more obedient than the androids in I Robotit will impale itself on a query that sucks every ounce of resources out of the system to return the data requested.

Managing Stored Procedures

I talked about stored procedures earlier, but I want to make sure that you understand how they can benefit client/server architecture. Remember, the idea behind client/server is to build an intelligent server and keep the client simple. Stored procedures are designed to help implement this strategy. Basically, a stored procedure is simply a program that runs on the server once invoked by the client (or another stored procedure). These can be written in T-SQL or in SQL Server 2005 in one of the CLR languages (as long as it's Visual Basic .NET or C#). Sure, these procedures can be very complexas complex as you likebut remember that this code must be executed by each client that needs the functionality, so it should be coded efficiently. SQL Server is very clever about loading and executing stored procedures, as I discuss in great depth in Chapter 5, so if several clients need the same stored procedure, the server can use a cached version already in memory.

By keeping the logic in the server, a client/server application can leverage its server-side code and not need to be redeployed when the stored procedure changesat least, if its signature does not change. The stored procedure signature is the set of input and output parameters (basically, the method parameters) required to execute the procedure. One could stretch this definition to mean the number and placement of rowsets and intermediate resultsets returned by the stored procedure. If these change, you're likely going to have to change the application that calls them as well.

I discuss invoking stored procedures in Chapter 11, but there are alluring ways to shortcut this process. A word of warning: Don't use these CommandBuilder-based approaches. They are expensive to execute at runtime and don't eliminate the need to redesign your code to deal with changes in the stored procedure logic.

Managing Client/Server Data Binding

When ADO.NET executes a query, it returns a rowset in the form of a DataReader. In a Windows Forms application, you can load a DataTable with a single line of code (as discussed in Chapter 12). If you "bind" the rowset to controls on your form, behind the scenes, ADO.NET moves that data from the rowset columns to the control's .TEXT properties. This means inbound data appears automatically in the TextBox or other "simple" bound controls (that show one row at a time) or "complex" bound controls, like the DataGridView control (that can show the entire rowset at once). In a Windows forms application, data binding is "full-duplex." That is, when changes are made to bound controls, the changes are automatically posted back to the ADO.NET DataTable (if you play your cards right). It turns out that in the 2.0 Framework, ASP.NET applications can also implement full-duplex data binding. The advantage of Windows forms data binding will be very clear once you start working with the Visual Studio 2005 IDE, which can automatically tie ADO.NET data structures (like DataSets, DataTables, and DataViews) to your form controls.

Over the last decade, Microsoft has spent quite a bit of time and talent to re-re-reinvent the VB3 Data control. This control ties a specific connection and rowset to one or more bound controls. In the pre ADO.NET days, you could position a pointer in the rowset (a Recordset in ADO classic), which reset the bound controls, posting changes in the controls to the rowset and moving data from the new "current" row to the bound controls. ADO.NET's data binding was a "start-from-scratch" effort that threw out this approach in favor of one that actually works. It seems that ADO.NET 2.0 has added yet another set of controls to the litany of ways to manage a bound rowset. As I discuss in detail in Chapter 12, developers can now choose to add a BindingNavigator to their forms to provide the scrolling features once provided by the Data control. Note that there is also a new ToolStrip control to capture the input parameters to the command being executed and provide a button control to execute the query to populate the DataSet. In this case, as shown in Figure 1.3, I'm executing a stored procedure with two input parameters.

Figure 1.3. Data binding in a Windows form application using the BindingNavigator.

[View full size image]

These new controls can make it easier to build Windows[17] applications that need to browse through rowsets row by row or move to the first or last row of the rowset. The BindingNavigator also permits an easy way to activate code to add or delete rowsthe IDE will even write this code for you if the SELECT statement is simple enough.

[17] Nope, these controls aren't available for web development.

Okay, but do these drag-and-drop controls really help when you have complex data structures returned from stored procedures? Nope, not really. These controls are fine for handling the rowsets returned from simple queries. When you start returning multiple resultsets or the product of a JOIN as the rowset, many of features exposed by these controls are left behind. Do you need to expose these controls to navigate through a rowset? No, they are pretty flexible, as they permit you to disable the parts you don't think are necessary, as well as supporting a number of visual layouts that I'll examine in Chapter 7, "The Visual Studio 2005 Data Tools."

Managing Server-Side Cursors

In earlier versions of ADOc (and RDO and DAO), developers could choose from one of several different types of server-side cursors maintained on a connection-by-connection basis. The most efficient of these cursors (the keyset cursor) permits the server to build a structure on the server that contains a pointer to each of the selected live rows (instead of returning physical rows). This means when the data row changed on the server, the client application can easily (and quickly) fetch the current row contents without having to run the entire query again (which could be prohibitively expensive). Server-side cursors only really make sense for client/server applications, as the cursor is "owned" by the connectionother architectures can't depend on a persistent connection. Yes, when the connection is closed, the cursor is dropped automatically. Unfortunately, early versions of ADO.NET never implemented server-side cursors and failed to do so again in ADO 2.0. The good news is that you can implement these yourself using ANSI SQL commands. The bad news is that if you aren't careful implementing server-side cursors, you can severely limit scalability and cripple system performance. An in-depth discussion on how to build ANSI server-side cursors is provided in Appendix IV, "Creating and Managing Server-Side Cursors."

Managing Temporary Tables

Because the server can save a temporary table on a persistent connection-by-connection basis, I've often created "helper" #temp tables (in TempDB) to shortcut JOINs and permit faster lookup table performance. Suppose a user is assigned to manage a particular part of the databasethe Washington and Oregon accounts. When this user logs on, I know his or her duties include only these two states, so I can execute a complex JOIN to fetch index rows that apply to this database subset because I know that subsequent operations need this list of applicable rows. When it comes time to actually fetch rows for this user, I use the #temp table I've created by incorporating it into a TSQL IN clause. Using this approach, I've been able to reduce query JOIN time dramatically. Remember that #temp tables live as long as the connection remains openwhen the connection is closed, the #temp table is dropped. That means this strategy works only if your application is designed around a persistent connectionnot one that's opened and closed frequently.

#Temp tables created within stored procedures are visible only within the stored procedure.

The Advantages and Challenges of Client/Server Architectures

As I've seen, there are several advantages to the client/server approach:

  • Carefully (correctly) designed and implemented client/server applications can be lighter "thin" clients that depend on intelligent server-side data validation instead of complex client-side logic that needs periodic maintenance and redeployment. The client-side logic can also be automatically morphed to deal with changing business constraints and rules.

  • The application can safely and efficiently manage and depend on server state. This includes the ability to create server-side cursors, as well as temporary tables that can be used to improve performance.

  • Client/server applications can afford to be more interactive and more functionally aware. That is, they can be more easily adapted to the Windows Forms application metaphor, which helps maintain application statethat can also mirror server state without extra coding.

  • Pseudo updates are far simpler and more practical, as the server is always within easy reach for testing data validationthere's no need to open a connection to perform background operations.

  • Applications can be written to support one or very few users and fairly easily adapted to multi-user situations.

  • The load on the network is extremely low. This makes client/server architectures ideal for low-speed connection conditions and permits a higher degree of scalability without over-taxing the network.

The challenges and disadvantages of client/server architecture include the following points:

  • Client/server architecture is best when you connect the clients to a server on a local-area network (LAN). Due to the connectivity strategies, this approach should not be implemented on networks subject to interruption in service. This means that it's not really suitable for use on the WWW. Sure, I've implemented client/server applications on ordinary modems, but I also included several layers of exception management that helped insulate the user from the vagarities of the connection.

  • Client/server applications are best when you're supporting a few to a few hundred users. As you add servers, you can scale to thousands of userson a LAN. If you expect to support the population of a small country, you had best choose another approach, like ASP.

  • A client/server application with a UI assumes you're going to build a Windows application. The problem here is that there are a dozen different versions of Windows in general use today and more on the way. Each version and each service-pack upgrade changes the environment in which you must run. When building a .NET application, you're going to have to ensure (or install) the correct version of the framework (at least) along with your application. Unfortunately, the framework does not run on all versions of Windows, so you have to ensure that your choice of architecture matches the target systemsbefore you commit to it.

  • Client/server architecture is not politically popular because of its ageit's been "done." While it can out-perform ASP applications in speed, features, and flexibility, it does not appeal to those that choose to connect over the web.

  • Traditionally, client/server applications have been far more expensive to deploy than newer ASP approaches. However, this problem has been addressed (at least, to some extent) with the new .NET "one-touch" deployment paradigm. Deployment is always a concern when installing anything on a user's computereven with low-impact deployment schemes. This makes client/server architectures less desirable for web-based systemsbut is not as big an issue for LAN corporate network deployment, where the company can more easily control what gets installed on the user's system.

ASP Browser-Based Architectures

No, I can't say that I'm an ASP expert. While I've learned a great deal about ASP architecture over the years, I don't pretend to know how to design (or tell you how to design) ASP or ASP.NET applications. Frankly, I depend on a number of technical resources for guidance in this genrelike Peter Blackburn. Remember, my focus is and has been on data access strategies, and ASP.NET applications also consume and update data. The low-level mechanics of how this is done are different than the strategies used in Windows Forms applications, but not that different. Consider that ADO.NET was designed (primarily) to support disconnected data structuresASP.NET applications have no need for pessimistic locking, server-side cursors or other techniques that require the server to maintain per-user/connection state. The connection-pooling mechanism was implemented to help ASP.NET applications open connections more quicklya Windows Forms application opens so few connections that connection pooling does not really help (if at all). ASP.NET applications also must handle security differently. Since many of these applications are designed to expose information and data interaction to the general WWW public, it makes no sense to try to manage these "users" with domain accounts, as you can (and often should) with Windows Forms applications. Sure, there are situations where you pass the user credentials through to IIS (which hosts ASP.NET), but very few applications take that route.

As far as handling data queries and updates in an ASP.NET application, there are a number of differences. Consider that in a scalable ASP.NET application, no (or very little) state is maintained on the server or in the structures passed to and from the browser (the View State). This means if your application executes a query, the rowset must be either persisted locally as a DataTable, DataSet, or TableAdapter object (which can be expensive), or requeried each time the rows are needed (which can be moreor lessexpensive, depending on the complexity of the query and number of rows returned). In previous versions, once the data is changed in the browser and the new data values from the controls are returned to your ASP page, you had to move the data control by control to the "bound" data table columns. In ASP.NET 2.0, you can set up full-duplex data binding so your code is simplerbut still not as simple as Windows Forms applications, which maintain a strongly bound link (in memory) between the UI controls and the underlying DataTable objects.

While there are differences, the design principles you use to build successful, high-performance, scalable Windows Forms applications are basically the same. Code your queries to return just the rows and columns you need in the immediate future. While scalable (and functional) ASP applications insist on an open-query-close connection strategy, this approach can also be used in Windows Forms applicationsbut it does not have to be. As you read through the rest of the book, you'll find additional hints and strategies that can help make your ASP application support more users with less hardware.

XML Web Service Architectures

One of the newest and most widely touted architectures released with the .NET Framework is XML Web Services. Basically, a Web Service is a block of code hosted on IIS (or an HTML Endpoint) that is exposed as a class. It uses XML structures to render its object names, methods, properties, and events. It exposes these structures via SOAP. Think of Web Services as a replacement for distributed COM (DCOM).

As far as data access strategies, this approach is similar in many respects to ASP.NET. Again, it's tough to maintain data state on a block of code that's designed to serve 100 or 100,000 users an hour. It's well beyond the scope of this book to do more than point them out as we tour other parts of the Visual Studio and SQL Server landscape.


Summary

This chapter tried to help bring a number of architectural options into view. Sure, it might have seemed like a bus tour through Rome where the driver only gestures out the window as he describes the 2000-year-old architecture in broken English, but I hope you got a pretty good understanding of the issues developers face when choosing a useful, scalable, and adequately performant architecture. I hope you understand by now that the choice of application architecture is gated by what you have the resources to dowhether those resources are skills, money, hardware, customer demands, or simply time. No, not every application you write needs to scale to support the population of Cleveland or even everyone in your company, but you should consider that if your application is successful, you might have to. In the next chapter, I'll get into more detail on which version of SQL Server makes sense, as well as show you a peek under the hood to see how SQL Server works.


Chapter 2. How Does SQL Server Work?

    Introduction

    Servers and Versions

    Installing SQL Server

    Running SQL Server Services

    Touring the SQL Server System Databases

    Understanding the SQL Server Security System

    Managing SQL Server Connections

    Using the SQL Query Tools

    Creating SELECT Queries

    Creating and Querying Views

    Implementing Business Rules

    User-Defined (Aliased) Types

    Implementing Constraints

    Managing Databases and Queries with Batches and Scripts

    The Query Optimizer and the Query Plan

    Understanding the Buffer Cache

    Executing Action Commands

    Introducing Stored Procedures

    Introducing Triggers

    Using Transactions to Protect Data Integrity

    Administrative Functions

    Working with the Transaction Log

    Summary


Introduction

From the very first Hitchhiker's Guide, I assumed that my readers understood SQL Server as well as the route they take to work every morning. They knew every shortcut and back alley to beat the traffic and how to bypass that speed-trap over by the lake. Based on questions from the newsgroups and those attending my conference sessions, it's clear that some of you would like to know, or need to know, a lot more about SQL Server's inner workings. In order to get the best performance out of any engine, you really should know what makes it tick and purr. No, I don't plan to get into the intimate details of the query engine and exactly how it chooses one index over another or how the file system writes data to the drive, but I do plan to provide you with the knowledge you need to get the most out of the query engine and optimize what gets written to disk.

Since SQL Server has captured about 50% of the market, a number of developers (perhaps you) are in the process of migrating from Access/JET, Foxpro, MySQL, or one of the other database management systems that you've outgrown. You might also be transitioning from one of the "enterprise"-class DBMSs, like Oracle, DB2, or Informix, or simply want to improve your SQL Server skillset. On the other hand, you might simply be ready to migrate from SQL Server 2000 or, if you work for the government, perhaps SQL Server Version 7.0 or earlier. This chapter outlines how to choose an appropriate version of SQL Server, how the server handles connections and security, how (in some detail) SQL Server processes the queries you submit, and how the data is returned. The information I provide here should help prevent you from being surprised at SQL Server's behavior, performance, or lack thereof as your application puts it through its paces. I can't hope to duplicate SQL Server's voluminous documentationI don't plan to try. I do hope that the points I raise make you aware of the base features afforded by SQL Serverespecially the new features exposed by SQL Server 2005.


Servers and Versions

While this book can only attempt to keep up with Microsoft's ever-morphing version configurations, it can try to outline the basic versions, highlight the differences, and note where each version is best applied to your application requirements. Yes, I touched on these issues in Chapter 1, "Application Architectures," when I suggested how to choose one version of SQL Server over another. In any case, I suggest you look on Microsoft's website[1] for up-to-the-minute details and prices. Basically, there are now six basic versions of SQL Server 2005and one version that only calls itself SQL Server, but isn't:

[1] www.microsoft.com/sql/howtobuy/default.asp

  • SQL Server 2005 Enterprise Edition: This is the most comprehensive version of SQL Serverand most expensive (it retails for $14,000 or so). It contains a comprehensive set of tools to create, manage, tune, and profile the DBMS. It also includes many features that large corporate production sites need. No, I don't expect a lot of you have access to this version unless you work in a large corporate (or independently wealthy) organization. This version is the only place that you can get the new Report Builder. No, this is not just Reporting Services, but an entirely new reporting interface.

  • SQL Server 2005 Developer Edition: This is a license-restricted version of the Enterprise Edition. It generally costs $50 or less and includes all of the tools and functionality included in the Enterprise Edition. This version cannot (by license) be used in a production environment. This edition is popular with developers (and authors), as it has all the tools they need to manage databases, users, and rights. Unfortunately, since it exposes functionality not available in other versions, applications developed against the Developer Edition might not work against the more affordable versions. Just be cognizant of the features you're accessing. The aforementioned web site has a chart that lists the specific features. I summarize that chart in Figure 2.1.

    Figure 2.1. Comparing versions of SQL Server.

    [View full size image]

  • SQL Server 2005 Standard Edition: Supports most functionality and tools included in the Enterprise Edition, except the high-end features such as Advanced Analysis Features for OLAP, Comprehensive Analysis Services, Distributed Partitioned Views, High Availability extensions, Virtual Interface System-Area Network, and Web-enabled analysis. It also does not have the new Report Buildera feature you might consider if you want your customers to write their own reportsPeter and I are not that thrilled with it. For those considering a way to manage and distribute reports on a more limited basis, you'll find that Reporting Services is hosted in virtually all versions of SQL Servereven the Expressbut not in SQL Server Everywhere. The Standard edition is limited in the sense that it supports only 4 CPUs but an unrestricted RAM cachebut it's more expensive than the SQL Server 2000 version: It retails between $2,800 and $3,000, but the Workgroup edition is cheaper than the old Standard edition. New for 2005 is the ability to mirror the database and automatically redirect a client in the event of database failure (post SP1). It also supports those of you who plan to use the new 64-bit processors from Intel and AMD. In this version, "passive" servers are free. That is, you can set up clustering, backup, and recovery servers at no additional cost.

  • SQL Server 2005 Workgroup Edition: This version is new for 2005 and pretty much replaces the Standard edition. It's meant to provide a lower-cost bridge between the free Express edition and the now pricey Standard edition. It's also cheaper than the old Standard editionretailing for about $740 to $3,900, depending on how you choose to scale the licenses. It's feature-limited like the Express editionit's held to two CPUs and 3GB of buffer cache, so it's designed to be a cheaper step up for those working with the Express edition that need more performance and scalability.

  • SQL Server 2005 Express Edition: This version is free, but it's limited even furtherto 1GB RAM cache and one CPU. SQL Server Express edition is designed to replace MSDE but leaves the thread governor behind. The only limit on performance is the buffer cache and single-processor support. It now includes a tools suite, better performance, and scalability. I'll take a closer look at this edition next. Note that SQL Server Express also changes how replication is supportedit's an important difference, so when it comes time to convert an existing MSDE-based system, be aware that there are new replication restrictions.

  • SQL Server 2005 ExpressAdvanced Services Edition: This version is identical to SQL Server Express, except that it comes bundled with Reporting Services, full-text search, and a customized version of SQL Server Management Studio. The real difference here is that it has a bigger footprintit's about 235MB on disk. This makes it harder (more expensive) to download and deploy.

  • SQL Server Everywhere: As I said in Chapter 1, this isn't really SQL Server, but it plays SQL Server in a soap opera. Well, not reallyit's really SQLCE, er, SQL Mobile Edition renamed. While it's very different from SQL Server, it can play an important role when your application needs a light, fast, easily deployable and encryptable database. I'll be providing a lot more content in my new EBook. Check the support site for details.

Since I expect that most readers are interested in the Standard, Workgroup, and SQL Express versions of SQL Server 2005, I don't plan to spend a lot of time talking about the advanced features afforded by the Enterprise EditionI'll leave that up to Microsoft. All of these versions (except SQL Server Everywhere) use the same core binary executables. They all execute the same T-SQL, use the same database files, and run the same server-side executables, including CLR-based stored procedures, functions, triggers, aggregates, and user-defined types. This means applications developed for the Express Edition can be easily upgraded to the Standard or Enterprise editions.

SQL Server Side-By-Side

One of the MSDN leads at Microsoft asked me to include my own side-by-side comparison chart that highlights the differences among the various editions. While the chart shown in Figure 2.1 might not be particularly scientific, it does put the versions into a slightly different perspective than the charts and sexy marketing material provided in the Microsoft documentation. Sure, the number of users supported is pretty arbitraryas you've seen so far in this chapter (and will find out later in other chapters), there are a lot of factors that gate how much work and how many users can be supported on a single SQL Server license. Of course, this number assumes that you choose to use the (more expensive) processor license or have purchased a sufficient number of Client Access Licenses (CALs).

All of these versions are supported by Visual Studioto one extent or another. In some cases, you'll have to revert back to SQL Server Management Studio to get some real work done. Don't throw away your toolset before venturing in to support SQL Server Everywhereit's also supported by Visual Studio and SQL Server Management Studio (to some extent).

Introducing...SQL Server Express Edition

In a SQL Server Magazine editorial published in 2003, Mike Otey hit a nerve when he said that "MSDE is free, but unsuitable for most multi-user projects." I don't think this is true. I followed up with an article that, for one reason or another, was never published. What I attempted to do in that article was to help to correct any misconceptions the readers (and Mike) had about MSDE and better describe what it is and why it's important to SQL Server developers. Well, clearly, it's too late to be talking about MSDEit's already morphed into the SQL Express edition of SQL Server. At this point, I want to compare MSDE and the new Workgroup, Express, and "Everywhere" versions to help everyone (even Mr. Otey) understand where these editions fit and what they are capable of doing, and understand their limitations.

I have been championing MSDE for quite a while nowwell before I left Microsoft in 2000. As a result, I feel a certain kinship to this particular version of SQL Server. I liked it because it provided a "free" alternative to JET and was a far better choice for small- to medium-size applications needing access to a professional, more scalable, and more secure DBMS. I saw MSDE (and now SQL Server Express) like the free cigarettes given to GIs in their C rationsbut without risk of lung cancer. These SKUs are designed to get developers at all levels comfortable with writing applications that depend on SQL Server's unique features. When they find that the free versions aren't enough, they have to come back to Microsoft for a fix on the real thing.

For years after its release, MSDE was an orphanit really didn't have its own product manager to champion it, protect it, and help educate the developer community. I wrote a number of articles about MSDE and continued to lurk on the public newsgroups in an attempt to keep the developer community informed about MSDE and how it can best be used. This was an uphill battle, as MSDE was far harder to install than JET and generally misunderstood thanks to countless "experts" telling developers that MSDE was limited to eight (or five) connectionsit's not.

All of that has changed. Microsoft and the SQL Server and Visual Studio teams have fully embraced MSDE in the SQL Server Express Edition (SSEE). Today, there's an entire team dedicated to SSEE. No, I'm not thrilled about how it's being "positioned" in the market, but as developers, we can use it as we think bestregardless of how Microsoft wants to position it. The problem (as I see it) is that SSEE is being targeted to so-called "hobbyists"people building "toy" applications. In contrast, I see SSEE as a viable SQL Server engine for small businesssuitable for use almost anywhere that JET is used today. It can support one to hundreds of users if the application is written carefully and can even be used to host data services on a web site. With the announcement of the Advanced Services Edition, SQL Server Express now supports full text search and Reporting Services. That puts it in the class of serious DBMS enginesit's more than a toy.

What Is SQL Server Express?

As I said, SQL Server 2005 Express edition is Microsoft's replacement for the MSDE Edition of SQL Server. It uses the same SQL engine implemented in SQL Server 2005 but comes packaged for use where far fewer users are expected to be servicedin smaller enterprises and smaller web sites. The biggest differences between MSDE and SSEE are in how they're installed and how they limit scalability. We'll see how SSEE's architecture has been completely redesigned to make its use more like the way you install and use Access/JET databaseswithout the deployment problems associated with MSDE. Since the new Advanced Services version of SSEE includes Reporting Services (for local data) and full-text search, you might find it's not necessary to upscale to Workgroup or Standard editions just to get those featuresat least, not right away.

SQL Server Express Edition has gone a long way toward building a better solution for applications that need:

  • A replacement for JET databases. That is, a DBMS that can be taken over by an IT department, if neededone that meets HIPA security requirements, uses all of the power of SQL Server to protect data and referential security, and does all of this regardless of how the user mistreats it.

  • A DBMS that can scale from a single user to several dozen users without requiring an upgrade to SQL Server Standard editionwithout having to worry about a governor degrading performance when it's needed most.

  • A DBMS that runs the same stored procedures, functions, aggregates, user-defined types, and extended stored procedures, whether they're written in T-SQL, in C++, or in a CLR assembly.

  • A DBMS that can work as easily on a small web site as in a client/server configuration.

  • A DBMS engine that can be installed easily and can be easily updated in place when service packs are made available. This means setup routines that are easy to integrate into your application's deployment scripts.

  • A DBMS that can be accessed by simply pointing to a DBMS file installed with or passed to an application in code. Because SQL Server Express is designed to permit databases to be attached on-the-fly, it's easier than ever to use "loose" SQL Server MDF database files and deploy them with your application. This makes it far easier to deploy a stand-alone SQL Server Express database .MDF file, as could be done with JET databases.

  • A DBMS that exposes a standard way to refer to a shared instance of SQL Server. When SSE is installed, by default, it's installed with the same instance name: SQLEXPRESS. This means your application's connection string can more easily target the SQL Server Express, whether it's installed on the local system or on the local area networkassuming the application setup routines take advantage of this feature. I'll talk about instance issues a little later.

What MSDE Issues Does SQL Server Express Edition Address?

While MSDE has been widely adopted by a large number of serious businesses, they often have to deal with a number of issues on their ownimplementing non-standard solutions that sometimes conflict with other companies' attempts to solve the same or other issues. These include:

  • Deployment: How is SQL Server installed along with an application? There is a litany of related issues here. For example, can SSEE be installed by a user, or do you need administrator's rights? What if SQL Server is already installed? What if the desired instance name collides with other existing instances? Should the application share an existing SQL Server instance or create a unique instance? What happens when the application that installed the shared SQL Server instance is uninstalledshould it also uninstall the SQL Server instance? If so, what happens to the other databases that instance is hosting?

  • Database deployment: How are databases installed on the target server? After installation, is the database visible to other applications on the server? How can applications hide proprietary data? How are Logins, Roles, and Users and their permissions configured?

  • Security: If you choose to share an instance of SQL Server, what password should be used for the system administrator (SA) account? How can user accounts be set up? Should the application simply use Integrated Security managed by the domain controller? What if there is no Active Directory?

  • Performance: MSDE uses a governor that limits the number of simultaneous operations on the server. What if your single-user application needs to perform several operations at once, but the governor kicks in and slows it down? Frankly, I'm not sure this issue is that widespread. I've heard very few people complain that the governor kicked in and made their application run too slowly. Sure, I've heard of applications not running particularly fast, but these were (generally) caused by brute-force queries or "challenged" database implementations. That said, I feel it's kinda onerous that you have to worry about a governor sitting there waiting to slow things down when things get busy.

  • Scalability: MSDE databases are limited to 2GB. What if you need to store more data than that? Does it mean you must upgrade your target systems to use a $1,500 Standard Edition SQL Server, which might cost more than all of the systems that expect to use it? Again, most of the complaints I've heard about in this regard are when people stored binary large objects (BLOBs) like documents or pictures in the database. Once they replaced the BLOBs with a path to the BLOB file, their databases shrunk down to a reasonable size. Consider that you can store about 4 million 500-byte rows in a 2GB database. Most medium- to small-business don't have quite that many customers, orders, and items.

  • Tools: The MSDE version of SQL Server is the "deployment" configuration. As such, it does not include any tools needed to administer the server or the databases it manages. I still recommend that developers buy the $49 (SRP) Developer Edition, which includes a full suite of tools to manage their MSDE databases. However, due to licensing restrictions, these tools cannot be deployed with an application. This means developers have to build their own client-side tools or simply build needed functionality into their deployed applications.

  • Administration: Regardless of how it's done, applications must take on quite a few administrative responsibilities. This is especially true for SQL Server Express systems where there is no "SA" or administrator at the keyboard. These admin responsibilities include managing the Login Account(s), permissions, backup, restore, and log maintenance. No, your end-user is not usually capable and should not be required or trusted to perform these operationsit's up to your application to do so. While JET databases needed periodic compression or repair, MSDE (and any SQL Server database) needs to periodically back up (and dump) the logs and database, but compression is done automatically. This issue becomes fairly significant when the database is not managed centrally where administrative and maintenance tasks can be more easily managed. Again, this is up to your application to do.

  • Service Packs: Since MSDE is often embedded in applications, users might not know that they have an instance of SQL Server installed. As such, they aren't aware that they might need to post a SQL Server service pack to protect their data and systems from attack even if they see it on the 5 o'clock news. To help prevent some of the problems caused by worms and other attacking viruses, MSDE SP3(a) disabled network connectivity so applications are unable to connect to the server over the intranet (or Internet). This issue notwithstanding, posting SQL Server updates to MSDE installations is problematic, as the Microsoft upgrades don't always work with custom setup scripts used to deploy MSDE applications and databases.

How Does SQL Server Express Address These Issues?

Developers, architects, and IT managers all over the world have been discussing and arguing about solutions to the aforementioned issues for several years. While there aren't solutions to all of these problems, SQL Server Express has addressed many of them by making some fairly fundamental changes. Before we get caught up in the differences, it's important to know what hasn't changed. SQL Server Express Edition is still free (with the usual licensing and use restrictions) and it still supports virtually all of the same features as MSDE except the ability to expose and manage a replication Publisher. Yep, this means that you'll need to pay for the Workgroup edition of SQL Server to be able to publish replicated tables.

SQL Server Express Edition with Advanced Services now supports Reporting Services, and it can be a Data Source for a server hosted on SQL Server 2000 Standard Edition. By default, setup still disables the ability to expose SQL Server Express instances to the network (as was first implemented in MSDE SP3), but visibility can be easily enabled. Let's take a closer look at SQL Server Express to see how it addresses each of the issues.

Deployment

SQL Server Express is designed to be downloadable over the web and installed on a user system just like any other systems-software. Yes, this assumes SQL Server Express is installed by the System Administratorthat is, unless you use the new User Instance feature that I discuss later in this chapter and again in Chapter 9, "Getting Connected." You can use the interactive Setup program (as I describe later) or run a command-line Setup executable. With "quiet" mode, the user need not see any SQL Server Express setup dialogs at alljust like the way you installed the MDAC stack and JET.

When you install SQL Server Express, by default, the Setup program attempts to create a common SQLEXPRESS instance. If it's already in place, you'll be given the choice of abandoning setup or choosing another instance name. The idea here is to get all applications that use SQL Server Express to share a common instancenot create one of their own. This makes application configuration easier (in some respects) and reduces the memory and disk footprint on the user system as well.

If you uninstall your application, it's also a good idea to uninstall any unique SQL Server Express instance your application installed. However, Microsoft recommends that you leave any SQLEXPRESS instances in place unless you're sure the system does not have any other dependant applications or databases that use it. One way to determine these cross-dependencies is to search the Master database for other databases that other applications might have attached or created. I'll show you how to create databases and dump the schema in Chapter 3, "Relational Databases 101."

Security

By default, SQL Server Express is configured to protect your data far more than MSDE (and certainly more than JET). As you install SSEE, you're given the opportunity to further tighten security or loosen it, depending on your requirements. One of the first decisions you'll have to make is to choose how the setup utility configures the SQL Server Express instance. An "instance" is simply an independent copy of SQL Server. Since Version 2000, SQL Server permits you to install several independent instances of the server executables. Each instance is treated like a separate entityeach has its own Master database, its own security configuration, and its own place on disk and in memory. When SQL Server Express is set up, each application (or you) must to decide whether it can coexist with other applications using a shared instance or whether it requires its own independent instance. There are security issues associated with each configuration, as I outline shortly. Note that SQL Server Express permits you to install up to 15 instances, but I don't expect folks to install more than 1 or 2 except in very (very) special circumstances. Remember, each instance consumes system RAM, so having more instances running can be very expensive.

Installing a Common Instance

By default, SQL Server Express assumes that you want to create (or use) a common instance named "SQLEXPRESS". Yes, you can also name your own "common" instance, but this assumes that all programs you install know this unique name. If you keep the default name (SQLEXPRESS), other applications can automatically share this common instance. With this approach, all databases are managed by a single shared Master database, and there is one System Administrator (SA) passwordwhich need never be revealed. When using a common instance, you might be able to see other installed databases, and other applications might be able to see your databaseunless you make sure that appropriate permissions are put in place. Generally, for home, hobbyist, or small office implementations, you don't usually have to worry about one application disturbing data in another database. If you install a single common instance, only one set of SQL Server DLLs, caches, and other memory-resident structures are loaded into memory. This means only one SQL Server instance consumes CPU resources.

Installing an Independent Instance

During setup, if you set the instance name to your own value, the installation program creates an entirely independent version of SQL Server Express. This instance has its own Master database; its own files, DLLs, and memory footprint; and its own SA password. Each independent instance starts a separate SQL Server service (program) that consumes memory and CPU cyclesin addition to any other instances that might be installed. While this approach gives you more security, in the sense that only those granted access to this instance can see the databases it manages, it's more expensive to implement and maintain. That's because each instance duplicates DLLs, caches, and other in-memory structures.

Installing the "Default" Instance

Another approach is to remove the instance name during setup. In this case, SQL Server Express is installed as the "default" instanceassuming a default instance isn't already installed. Only one server instance can be installed as the "default" instance. Again, if this is the only instance installed on your system, there is very little difference between the other configurations except when it comes time to connect to SQL Server Express, as I discuss later in this chapter.

Activating the User Instance

A new feature for SQL Server Express is the ability to open a Connection using the "User Instance = True" key. In this case, SQL Server and the SqlClient provider work together to create a private user-specific instance of SQL Server in the user's Documents and Settings folder hierarchy. This instance is given its own Master, Model, and TempDb, as well as rights to be accessed by the user as SA. I discuss this approach (which is fraught with issues) later in Chapters 3 and 9.

Choosing the System Administrator's Password

The System Administrator (SA) password is the key that unlocks the entire database. The SA is permitted to do anything with or to the databases or the information they contain. The SA can add, change, or remove databases all without anyone knowing the changes have been made. It's critical that this password be set correctly and protected. By "correctly," I mean that the password must be complex enough to prevent anyone (or any program) from easily guessing the value. No, "password" is not a good choicebut it's better than nothing.

When you install SQL Server Express using a common instance, there is only one SA password to worry about. Since the SA account is accessible only when you choose to install using Windows authentication, the SA password need never be revealed. In any case, when you install SQL Server Express and specify mixed-mode security, you're asked to provide an SA password.

Microsoft (and I) recommend that you configure your SQL Server Express instance to use Windows Integrated Security authenticationat least, at first. This means the computer and Windows domain system administrator accounts are granted full "SA" access to the SQL Server Express instance. Sure, you'll need to be a computer or domain administrator to perform maintenance and install databases, as well as perform operations as simple as changing the database table values. No, this does not mean everyone who uses SQL Server Express should be an administrator. It does mean that as part of the setup regimen, you'll need to create a "user" or application Login and set appropriate permissions on the tables, views, functions, and stored procedures that your application needs. I discuss this in more detail later in this chapter.

Performance

SQL Server Express has abandoned the concept of a "governor." Frankly, I've rarely seen the governor slow down any MSDE system, but by dropping the governor, Microsoft has removed a point of confusion about the scalability of the SQL Server engine. Yes, SQL Server Express has ways to limit scalability and performance. SQL Server Express is programmed to address only 1GB of available system RAM for its buffer pool. This limits the number of data pages and procedures in the cache. Any professional can tell you that the easiest way to improve performance to SQL Server is to add memory to the system, which automatically permits SQL Server to allocate more memory to the cache. Limiting usable RAM cache to 1GB means that you'll (eventually) cap performance as you add load to the SQL Server Express instance. Does that mean SQL Server Express can support 1,000 users? Sure, if the load (diverse operations/second) placed on the SQL Server Express instance is not that great. In the same way, 10 users could bog down any version of SQL Serverespecially if the application is not written efficiently. Consider that the RAM cache does not have to be very large to hold all of the pertinent data and procedure pages if the application is focused and the database is small.

SQL Server Express is also limited to a single processor instead of being able to run threads on additional processors if your system supports it. This limitation also tends to scale back the upper limit of the performance you can expect from SQL Server Express. Again, with efficient design, your application should still be able to scale to several dozen users without problems.

When an application using SQL Server Express ends, SQL Server does not shut down. No, there is no "autoshutdown" option in the SQL Server Express version. Because of this, the SQL Server engine is left in memory and continues to consume system RAM and CPU resources even after your application has ended. Yes, it's possible to write SQL Management Objects (SMO) or other (easier to write) routines to shut down the SQL Server Express instance, but this needs to be done only when you're sure it's not being shared by other applications.

Scalability

While MSDE databases are limited to 2GB, SQL Server Express database files are capped at 4GB. This means you can store twice as much data as beforethat's over 8 million 500-byte rows. Frankly, why some small companies think this is a big issue puzzles me. I've worked with large corporate and small city mainframe databases that fit nicely on a single 40MB disk packalong with the data logs for our Star Trek game. I guess people like to use the database to store a lot of documents and pictures of their pets. If you eliminate the BLOBs, SQL Server Express can support most medium-size business databases. As with MSDE, the log file size in SQL Server Express Edition is not limitedat least, not artificially. You still need to back up and truncate the logs periodically, as I discuss later.

Replication

One of the more important issues (and serious limitations IMHO) of SQL Server Express edition is the inability to publish replications. That is, SQL Server Express edition can be a transactional or merge replication subscriber, but not a publisher. MSDE, on the other hand, supports both snapshot and merge replication publications as well as subscriptions. Given the importance of replication architectures and how SQL Server Everywhere is likely to use replication to sync its database, having to buy the Workplace edition to host a replication publication seems, well, expensive.

Crippled Features

There is another set of features that Microsoft crippled in SQL Server Express editionthey didn't want to give too much away. These include Query Against Views, Indexing XML datatypes, index performance improvements, parallel service security, failover clustering, log shipping, database mirroring, high-availability maintenance features, and rich status reporting. Sure, some of these features will impact performance, but others won't really be missed until you start working with larger, enterprise-size database systems or have to support many more users.

Tools

Microsoft has changed its approach to the tools as well. Even if you don't count the new GUI setup, when you download SQL Server Express, a new version of OSQL, SQL Computer Manager (MMC snap-in), and SQLCMD command-line tools are included to help manage the SQL Server Express instance. In addition, Microsoft has retooled its SQL Server Management Studio to accommodate the limitations (and features) of SSEE. It's available via separate download called the "SQL Server 2005 Express Edition Toolkit," which contains these tools along with Reporting Services. I think you'll find that to get all of the free features of SSEE, you'll have to download three or four separate packages. I'll show you how to install all of those later in this chapter.

Administration

Yep, all of the tasks you had to do to administer MSDE also have to be done with SQL Server Expressjust as they have to be done with other versions of SQL Server. I would love to see an automated log backup script that periodically dumps the database and logs, and then truncates the log. Perhaps that's something that some enterprising third party needs to create. Until then, I recommend developers build these admin tasks into their applications and use SMO or the new framework factory classes to perform these needed maintenance functions and use Windows Scheduler to help. Again, I'm writing a new EBook on these database admin classes.

Service Packs

SQL Server Express can be installed only using the Windows Installer (MSI) installation package files. Unlike MSDE, you won't be able to create custom MSM setup scripts. In other respects, upgrades are done the same as in SQL Server Standard Edition, so you'll need to update the SQL Server engine via traditional Service Packs. Microsoft is acutely aware of the issues in this regard and is still formulating a better strategy.

What About the Workgroup Edition?

The newest member of the SQL Server family pushes the Standard edition up the SKU tree, so it would be the next logical step up from the Express editionat least, price-wise. The Workgroup edition should perform better than the Express edition, as it supports three times as much RAM cache (3GB) and two CPUs instead of just one. It also supports more replication scenarios, but not as many as the Standard edition. It can now host Reporting Services, but as I'll show you in Chapter 14, there are new reporting options afforded in the "lesser" editions. The Workgroup edition does support log shipping, Data Transformation Services (to import and export data), and limited replication publishing.


Installing SQL Server

While some of you will use established SQL Server installations, far more of you will install SQL Server on your local development machine or (better yet) on a dedicated system configured to host the SQL Server service. I often resurrect an old clunker to host SQL Server. Because the system might not have a lot of performance, I often get a more realistic idea of how a query performs under load. I also access SQL Server the same way I would in production (over a network), and my connection parameters and security configuration are virtually identical to the way the production server is accessed. It also means that if I barbeque the server, it won't make the DBA (and the rest of the team) mad.

IMHO

Your DBA won't mind nearly as much if you melt down your personal SQL Serverinstead of the production system.

This section walks you through the process of installing SQL Serverin this case, the Express edition, as its setup is very similar to the Workplace, Standard, and other "paid-for" versions. It's actually a bit more complicated, as, in order to get all of the "free" functionality, you have to run setup at least three timesonce for the SQL Server Express, once for the Advanced Features options, and again for the SQL Server Express Edition toolkit (which contains Reporting Services and SQL Server Management Studio Express).

Unlike MSDE, which does not support any form of a GUI Setup utility, SQL Server permits both command-line setup and an interactive GUI version that will be familiar to developers who use the Workgroup or Standard (or better) editions. The Express edition is a 53.8MB download available from Microsoft's MSDN downloads areas on the web. At this same location, you'll find separate downloads for the SQL Server Express Management Studioits custom toolset, as well as the Advanced Services SKU that includes everything (SQL Server Express, the tools, Reporting Services, and full-text search). The Advanced Services edition's download is 234MBconsiderably larger and more difficult to download on demand.

Once you start the SQLEXPR.EXE (or the SQLEXPRE_ADV.EXE) setup program, your system is examined to see if you've paid your light bill and if you have the 2.0 .NET Framework installedyup, it's a prerequisite. At this point, the Microsoft SQL Native Client and setup support files are loaded (or validated). All other dependencies and system minimum configuration requirements are then scanned, a system configuration and conformity check is made, and any missing pieces are installed. This is all done invisibly behind the scenes in the "quiet" install. Quiet installs are launched from a batch or command-line prompt using syntax similar to that shown in Figure 2.2 (no, this is not necessarily the name of your setup executable).

Figure 2.2. Launching the setup program using the "quiet" option.

[View full size image]

You'll also want to revisit the SQL Server downloads page and pull down the latest Books Online for SQL Server Express (and other editions).

Early in the setup process, the SQL Server Express GUI Setup program exposes a dialog (shown in Figure 2.3) that asks if the user wants to hide the "advanced configuration" options. (Don't confuse these with the "Advanced Services" SKU.) These advanced services refer to alternative security configurations. By default, Setup configures the SQL Server Express instance being installed to use Integrated Security and disable all access to the TCP ports and external protocols. This means you won't be able to access the SQL Server Express instance from other systems or by using SQL Server credentialsunless you change the advanced configuration options. This also means you won't be able to use SQL Server Express as a network SQL "server"just as a locally accessible DBMS that only the client application can use. Note that setup problems are logged to the system event log.

Figure 2.3. Enabling advanced configuration options.

Choosing Advanced Services Options

If you're installing SQL Server Express with Advanced Services, you're confronted with another dialog. It's wise to drill into all of the grayed options to ensure that all of the options you need are checked, including Replication, Full-Text Search, and the shared tools. Notice that Reporting Services is not shown in this listit's included when you install the SQL Server 2005 Express Edition Toolkit (SP1)a separate download. No, for the life of me, I don't know why these weren't combined. Note as well that you have the option of installing selected tools, including the SDK and the new custom SQL Server Management Studio Express, as shown in Figure 2.4.

Figure 2.4. SQL Server Express with Advanced Services options selection dialog.

Setting the Instance Name

If you chose to enable Advanced Configuration options, the next option dialog you'll see prompts for the SQL Server Express instance name. Figure 2.5 illustrates this dialog. No, I don't recommend that you change this name, but yes, with a little luck, it can be changed later.

Figure 2.5. Choosing the instance namethe default is SQLExpress.

Choosing the Service Account

The next dialog prompts you to name the Service Account, which determines which credentials are used when SQL Server support services (like full-text search and Reporting Services) logs into SQL Server. I usually create a dedicated machine account (SQLE) for this purpose. It gives me the flexibility of granting or denying specific rights to these operations. Setup automatically alters the rights on this account to accommodate its needs.

Figure 2.6. Setting the Upgrade Login Account.

Setting the Authentication Mode

The next dialog you'll see (if you chose the Advanced Configuration options) determines whether the SQL Server Express instance should recognize SQL Server authentication credentials"Mixed Mode" security, as shown in Figure 2.7. This option is disabled by default, so if you want to use SQL Server user-id/password authentication, you must enable it here. If you choose to enable mixed mode, you'll also have to specify a "strong" (non-blank) SA password. Again, choosing mixed-mode authentication opens your server to further points of attack. In any case, SQL Server always accepts validated Windows accounts when connection attempts are madeassuming there are appropriate SQL Server login IDs created. I discuss this further in Chapter 9.

Figure 2.7. Setting the SQL Server Express Authentication mode.

It's a good idea to leave Mixed Mode disabled until you understand the security implications of using SQL Server security, as it forces developers to figure out ways to hide the SQL Server credentials used by their application to prevent their use by unscrupulous hackers. Even then, it's still a good idea to stick with the SSPI (Windows authentication) unless your design makes this configuration impossible.

IMHO

Where do hackers come from? A recent study said that most "hackers" (those trying to gain unauthorized access to protected data) were individuals within the corporate firewalland most (by far) were on the company payroll.

While the SQL Server Express Setup utility permits you to set the type of security used by the SQL Express instance, you can also alter this or other settings by using the new SQL Server 2005 Surface Area Configuration Tool. As I discuss in more depth later, the default security mode is "Windows Authentication", which validates user credentials against the Domain's Active Directory database.

Setting Collating Sequences

Unless you live outside the U.S., you won't have to configure the collating sequences, but if you do, now's your chance. These settings determine how SQL Server sorts and locates values when making string comparisons. It also determines whether or not SQL Server's T-SQL syntax is case-sensitive. I heartily recommend that you not change the default (non-case-sensitive) setting unless you have a letter from your mom and the entire team understands the implications.

Enabling User Instances

The User Instance feature that I briefly mentioned earlier is enabled by default. If, for some reason, you don't want to permit applications to create user instances of SQL Server Express (and only SQL Server Express can create these unique user-specific instances), you can disable it with the next dialog. Frankly, this is a big step, so don't enable User Instances until you're sure that it meets your setup configuration and security requirements.

Finishing SQL Server Express Setup

Several more (intuitive) dialogs prompt you for feedback and other less-important information that I'm sure you can figure out. After that, the setup program takes off on its own and won't ask any more questions until it's done. That alone is a great improvement over older versions that would stop well into the setup process and ask to proceed so they could finish.

Once your SQL Server Express instance is installed, the SQL Server Surface Area Configuration Tool (shown in Figure 2.8) or SQL Server Configuration Manager (MMC snap-in) can enable the TCP ports or appropriate network protocols. These tools are very intuitive, so I won't bore you with a house-by-house tour. If you want to make your SQL Server Express edition visible to the network (you probably should think about this before doing so), you'll also have to start the "SQL Browser" service to provide server name resolution and enable the needed protocols.

Figure 2.8. Using the SQL Server 2005 Surface Area Configuration Tool.

[View full size image]

IMHO

Remember, the "Slammer" worm exploited the fact that most SQL Servers are exposed on UDP port 1434. This means SQL Server Express won't be prey to this type of attack unless you enable the SQL Browser service.

Yes, there's also the (fairly crude) SQL Server Configuration Manager, which is designed to perform many of the same operations as the Surface Area tool and a few more (like inspect and configure specific TCP ports). This tool is an easy way to discover which TCP/UDP ports have been assigned to each SQL Server instance installed. As you can see from Figure 2.9, it can be used to start, stop, pause, and restart/resume any instance of a SQL Server or the support services.

Figure 2.9. Using the Computer Manager MMC snap-in to manage installed services.

[View full size image]

Once Setup is completed, the setup files (which can contain plaintext or weakly encrypted credentials and other sensitive configuration informationbasically, the keys to your server) should be deleted or secured.


Running SQL Server Services

Unlike JET, SQL Server Everywhere, or other DLL-based DBMS engines, SQL Server runs as a Windows service. Reporting Services and several other SQL Server extensions also run as additional, separate Windows services. This means you'll have to ensure that the specific SQL Server instance service you wish to access is running before trying to open a connection. This section discusses the ways you can inspect and configure the service(s) installed by SQL Server setup and set them to run whenever Windows starts or on demand. Depending on how the intended "server" system that hosts these SQL Server services is used, you might find it necessary to start and stop SQL Server programmatically instead of letting it sit there idle chewing up resources like a visiting in-law lounging on your sofa watching your TV and eating your last bag of Fritos.

Inspecting SQL Server Services

There are a variety of tools included with SQL Server that permit you to determine the visibility and state of any of the SQL Serverrelated services. Figure 2.10 illustrates the SQL Server Configuration Manager utility that exposes (just) the SQL Server 2005related services. In this case, the utility has found several instances of the SQL Server service, as well as other support services like SQL Server Reporting Services, FullText Search, Integration Service, and Analysis Services, as well as the SQL Server Browser and SQL Server Agent services. Ah, no. This installation is clearly not just a SQL Server Express edition rig.

Figure 2.10. Inspecting the installed SQL Server 2005related services.

[View full size image]

Notice that only the SQL Server Browser and FullText Search Services are started (indicated by the green arrow); the SQL Server Browser service is stopped. This tells me that I won't be able to open any connections to any instance of SQL Server on this systemnot until I start one of the instances. I often leave my demo systems configured like thisenabling just the features that I needhopefully, before you show up to the session.

The SQL Server Configuration Manager can also be used to manually start, stop, pause, or resume the service, as shown in Figure 2.9, or set the Start Mode of each service by right-clicking on the service and choosing the appropriate running state or Properties | Service to set the Start mode, as shown in Figure 2.11.

Figure 2.11. Setting the Start Mode for a specific SQL Server service.

IMHO

The SQL Server Configuration Manager can be as contrary as my grandkids at bedtime.

I have not always had consistent results with this application. It seems to get confused sometimes and shows the service state incorrectly. If this occurs, stop and restart the tool or use the Services.msc MMC console application to manage your SQL Server services. Frankly, since the MMC console application also exposes SQL Server 2000 instances, it might be a better choice anyway, if you know how to use it.

Starting a SQL Server Service

One thing the setup routines don't include (which I think they should) is the ability to determine whether or not a SQL Server instance should be automatically started during system startupby default, SQL Server Express instances are not started. In contrast, other SQL Server versions are configured to start automatically when Windows starts. As I said, if you try to connect to any server instance that's not started, your application (or tool) will receive an exception message that says the server cannot be found. If you want to avoid this problem, you're going to have to make a decision:

  • Set up the SQL Server instance to start each time the system starts. In this case, your client system has to bear the memory and CPU load of SQL Server at all timeswhether or not your application is running. This might not matter at all if the system is dedicated to this application. This option is required if you plan to use SSEE as a server.

  • Add code to your application that starts the SQL Server instance of your choosing before trying to connect to it. I think this is the more viable approachespecially if you know that yours is the only application using the instancebut not if the SQL Server instance is going to act as a central server for other users. Given that the SQLExpress instance is designed to serve as a "common" instance, it might be dangerous to arbitrarily shut down the instancebut it should not hurt to start it. I talk about how to start an instance of SQL Server programmatically in Chapter 9.

If you choose either option, use the SQL Server Configuration Manager to set the startup behavior (Start Mode) to "Automatic" or "Manual", as shown in Figure 2.11.

Using SQL Express as a Server

Sure, it's possible to configure a SQL Server Express instance (or any instance of SQL Server, for that matter) as a shared server that anyone on the local area network (or WAN, for that matter) can see and access. While I'm not a fan of exposing SQL Server on the Internet, it can be donebut at your own peril. Even SQL Express is capable of supporting hundreds of users in a LAN configuration if the demand per user is not that high. If you choose this route, you'll need to be sure to set up the ports and protocols to permit the server to see the LAN and to enable the SQL Browser service as described earlier. You'll also have to make sure that the system hosting the "server" instance is availablethat is, it's not locked away in someone's office where it can't be started or reset if something goes wrong. Of course, you'll also need to ensure that the instance Start Mode is set to "Automatic", as shown in Figure 2.10.

Remember that SQL Server is optimized through judicious use of RAM and CPU resources. If the system hosting SQL Server is also required to handle other duties, such as acting as a print, email, or IIS server, it won't be able to handle nearly as many users.

Any version of SQL Server can also be accessed by IIS and ASP applications.

Connecting (Gaining Access) to SQL Server Service Instances

Microsoft and I suggest that you learn to break any dependencies on existing COM and the OLE DB providers in favor of using the new "native" SQL Native Client (SNAC) provider when accessing SQL Server from Visual Basic 6.0 or other COM-based applications. This new OLE DB provider does not require installation of the MDAC stack and exposes all of SQL Server 2005's new datatypes and many other features, like (gulp) MARS. Sure, when using ADO.NET, the SqlClient.NET Data Provider is still your best choice when accessing SQL Server. Yes, if you must connect to SQL Server Express from COM-based applications using MDAC and OLE DB, you can do so by using the old SqlOleDB or ODBC providers, as well as the new SNAC provider. Be aware that there are some subtle differences between how the old SQLOLEDB and SNAC providers workespecially when it comes to handling parameters.

ADO.NET also supports a way to access and automatically attach a selected database .MDF file. When working with a JET database, all you needed to do as you deployed your application was to copy the database .MDB file to the target client system. SSEE now supports a similar technique by exposing a connection string option (AttachDBFilename). This lets you point to an .MDF database file that's attached to the master on first open. This makes it very easy to deploy an SSEE database. I'll discuss this option when we start talking about opening connections using ADO.NET in Chapter 9.


Touring the SQL Server System Databases

This section discusses how SQL Server manages itself and the databases installed thereon. All versions of SQL Server[2] implement the same (basic) core functionality and depend on the same four system databases. It turns out that SQL Server uses these internal databases behind the scenes to manage all other databases installed on the server. Using these databases, SQL Server keeps vital Data Description Language (DDL) information about each database, login, and every other database object in the system and user databases. This section takes a closer look at these databases, as well as how they are exposed to the network or your local application.

[2] Remember, "SQL Server Everywhere" is not really SQL Server at all.

SQL Server Instances

Several separate SQL Server "instances" can be installed on a single systemlimited only by your licenseeach of these separate installations is referenced by name using a standard syntax of "<Machine Name>\<Instance Name>" in addition to the default instance, which is referred to by just the machine name. Yes, each of these instances consumes space on your hard drive and (considerable) space in RAM, so, no, it's not a good idea to install too many instancestwo or three should be more than enough for most systems.

I took questions at a newsgroup where the members asked if they could get around the Express Edition limitations by installing another instance (or several instances). I said, yes, it's possible to have a single system handle more load using more than one instance, but it's a pretty silly way to get around the issue. It would make more sense to dedicate several systems to the Express edition instead of loading up a single processorbut even that's a bit silly. The Workgroup edition costs less than another system.

If you don't name a SQL Server instance but choose to configure the "default" instance, SQL Server's name is set when you build your computer systemit's the Windows machine name. While it's easy to rename your machine, remember that SQL Server won't know you've changed its name until you tell it you've done so. Figure 2.12 shows you how.

Figure 2.12. Telling SQL Server your system has been renamed.

Multiple Instances or Multiple Databases?

Each server instance can manage up to 32,767 databases, but generally, I don't ask a SQL Server instance to support more than a few databases, at most (and usually only 1 or 2) for performance reasonshardware is simply too cheap. Multiple instances are useful when it's necessary to create an application that depends on a dedicated SQL Server managing a unique database that can best be controlled by the application. Multiple databases on a single instance can be more efficient, as the server can more easily share resources (if they exist) between databases. While the engines can pass resources between "linked" server instances, it's pretty expensive to do sobut it can be an effective way to execute JOIN operations across diverse DBMS platforms. See "Linked Servers" in BOL for more details.

If you create an application that needs to store data on the local client workstation, you can include a quiet SQL Server (Express or other versions) setup in the application setup routines so the server instance is installed right on the client's system. Since there might already be an existing instance of SQL Server installed on the system, your application setup logic can choose to create a separate instance of SQL Server that does not affect the current instance(s). This new instance gets its own set of SQL Server support databases stored in their own files (see Table 2.1) and its own system administrator (SA) password. This also means you can back up, restore, or uninstall your application and SQL Server instance without disturbing the other instances.

Table 2.1. System and Application Databases Installed by SQL Server Setup

Database

Purpose

Default Size

master[3]

To manage the database descriptions, security login information, and other system-level information about the databases stored in this SQL Server instance. Also include here are stored procedures used to manage and protect all databases. Master is used to store all Login account and password information. It must always exist on any SQL Server instance.

11.0MB data, 1.25MB log

model

Used as a template for all new databases created on this SQL Server instance. The model database is re-created each time SQL Server starts. It must always exist on any SQL Server instance.

0.75MB data, 0.75MB log

tempdb

SQL Server's "scratchpad". It's used to hold temporary tables and stored procedures, as well as work with tables, cursors, and sorted rowsets on a connection-by-connection basis. It's also possible to create "global" temporary tables here that are visible to all connections. The tempdb database is re-created each time the SQL Server is started (from model), so nothing remains in tempdb between sessions. As connection sare dropped or "reset," the contents of tempdb are purged of anything created by the connection so applications should not see anything left over from a previous connectionunless you disable the "reset connection" feature. (I show how to do this in Chapter 9.)

Built from model

distribution

Used by the replication components of SQL Server, such as the Distribution Agent, to store such data as transactions, snapshot jobs, synchronization status, and replication history information. Any server configured to participate either as a remote distribution server or as a combined Publisher/Distributor has a distribution database.

Varies

Msdb

The msdb database is used by SQL Server, SQL Server Management Studio, and SQL Server Agent to store data, including scheduling information and backup and restore history information.

4.69MB data, 0.09MB log

ReportServer

Reporting Services catalog. Used to manage the reports and all aspects of Reporting Services.

At least 3.88MB data, 1.04 log

[3] Notice that these database names are all in lowercase. This is relatively unimportant unless you install your DBMS as "case-sensitive."

The recommended approach is to leverage the SQL Server Express (SQLEXPRESS) instance, if it exists. This reduces the complexity of the install and imposes less of a burden on the client system. Remember, each instance consumes memory, CPU time, and disk space.

Remember that if the User Instance feature is enabled and your application requests this option, separate and independent support databases are copied to the current user's Documents and Settings folder hierarchy. This approach permits applications to access the "user instance" of SQL Server as SA using the user's Windows credentials. I discuss the mechanics of this approach in vivid detail in Chapter 9.

User and System Databases

There are two types of databases managed by SQL Server"system" and "user". Both can be managed in varying degrees by SQL Server using the UI-driven tools such as Visual Studio, the SQL Management Studio, and your own code using .NET Framework classes and SQL Server Maintenance Objects (SMO). User databases contain your data as well as the support infrastructure used to protect, index, and store the data. System databases are created automatically when you install or start SQL Server, as described in Table 2.1.

Creating a User Database

It's not hard to create your own SQL Server user database using the tools that come with SQL Server. Check out SQL Server Management Studio (or the Express edition's SQL Server Management Express) and right-click on "Databases" in the object explorer. Choose "New Database..." to create a new user database, as shown in Figure 2.13.

Figure 2.13. Creating a new SQL Server database using SQL Server Management Studio Express.

All versions of Visual Studio also support the ability to create new SQL Server databases, as I describe in Chapter 5, "Managing Executables with the Server Explorer"some have more choices than others. In most cases, creating a new user database is as easy as creating a new Database project or Data Source and feeding answers to the wizard. While I rarely find the need to use the SQL Server dialect of SQL (Transact SQL [T-SQL]) to create databases, if you find a compelling need to do this and want to impress your friends, check out the T-SQL statement CREATE DATABASE in Books Online or Chapter 3 where I describe how to create your own databases and populate them with user objects like tables, views, and stored procedures.

When your server starts, it copies the model database to tempdb. That's fine, except when you have queries that rely heavily on tempdb. For example, when you use ORDER BY to sort the rowset, SQL Server uses tempdb as a scratchpad. If it has to stretch the size of tempdb as it builds the sorted rowset, initial performance can suffer. To resolve this issue, determine the size of tempdb after your system has run the procedures that use it. Next, set the size of the model database to make sure tempdb starts at this larger size.

Managing User Databases

If you use SQL Server Management Studio, you'll see that the System databases are broken out from User databases on each instance of SQL Server to which you connect. You might also see two ReportServer databases listed among the user databases. Depending on which sample databases you install (and this book's DVD includes many of these), the list of user databases might be quite long.

Once you attach or otherwise create your own user databases (as I'll show you in Chapter 3), these are logged into the master database. At this point, these user databases become "known" to SQL Server and visible to authorized login accounts and T-SQL queries. Each database stores its own DDL schema information, but the master database contains any "global" properties that apply to all databases and the security credentials required to access them. Whenever you add a new database or change the schema of an existing database, be sure to back up the master database soon thereafter. This will help when you need to rebuild your server after the flood, fire, or visiting in-laws. Before talking about creating user databases, I need to touch on how SQL Server protects itself and the data you ask it to store.


Understanding the SQL Server Security System

Security is one of SQL Server's most important jobsand it should be one of yours as well. If you aren't careful or don't follow some basic rules, you can easily disable much of this functionality and expose your data to nefarious eyes or clumsy programmerspossibly to the entire network or the WWW.

Traditionally, SQL Server gains much of its performance by "physical" security, that is, by restricting access to the physical server machine either by locking it away in a limited access room or by carefully managing access to it in other wayslike using a moat populated with underfed alligators. Once a nefarious person has physical access to the SQL Server database .MDF file, it can be copied to another system and accessed at their leisure. Of course, if sensitive data columns are encrypted, the evil-doer won't be able to access anything vital.

SQL Server's software-based security system has two outer layers:

  • Authentication, which establishes the identity of an individual (or account) attempting to open a connection or access an object (a database, table, view, stored procedure, or any other object). Login names and their credentials are used to manage "who" can gain access to the server. In some cases, Windows can capture this login name and user credentials. In other (fairly common) situations, these values are captured or generated and provided by the application itself.

  • Authorization, which is the mechanism SQL Server uses to determine if a validated identity has rights to access, update, or delete any given object. Usernames are used to validate rights on databases and the objects they contain.

SQL Server's security system is fairly complex, and it's well beyond the scope of this book to attempt to explain it allnot that anyone on the planet besides Kalen Delaney[4] actually understands it all. In basic terms, SQL Server uses authentication schemes to gate access to the database engine and the objects it manages based on login credentials and by exposing (or disabling) specific TCP and UDP ports to the network.

[4] Kalen Delaney is an old friend and invaluable resource for the SQL Server community. She's written a number of articles on this subject, the first of which can be found here: www.winnetmag.com/SQLServer/Article/ArticleID/40963/40963.html.

As I've just discussed, the default installation of SQL Server Express Edition doesn't expose its ports to the network, so these must be activated to be able to access this version from other computers.

Addressing Objects in SQL Server 2005

When you start digging into SQL Server 2005, you're bound to notice that the term schema is used in a new way. For decades, I've used the term schema to mean the data model, how the data tables are related, and how the database is designed. In SQL Server 2005, Microsoft has implemented a feature that (according to a credible source[5] at Microsoft) was specified in the ANSI SQL-99 specificationthe ability to address objects by "schema" and not owner. So what is this "User Schema Separation in SQL 2005" all about? Rather than explain in depth what it is, let's first take a look at how objects are addressed in SQL2000:

[5] Pablo Castro is another friend at Microsoft who helps us from time to time with sticky ADO.NET issues.

1.
Create a User account in SQL Server 2000 (TestPerson)it can be either a domain account or a SQL Account, although you'll find it more convenient to create it as a SQL Account for this demonstration. Figure 2.14 shows the dialog used to set up this account.

Figure 2.14. Creating a new login with SQL Server 2000 SQL Enterprise Manager.

2.
Next, choose a database. For purposes of this demonstration, I chose biblio[6], but you can use any database where you have permission to add tables. Make sure the account has access to the test database. No, don't provide the TestPerson account with dbo rights to the chosen Database, but the account does need permission to create tables, as shown in Figure 2.15.

[6] See Appendix I, which steps you through the process of installing the sample databases from the book's DVD.

Figure 2.15. Setting the database access rights for the TestPerson.

3.
Next, as the dbo or Sysadmin user of the target database, create a table using the DDL, as shown in Figure 2.16. Using SQL Enterprise Manager, you'll see that the table created is actually created as dbo.A. No surprises here.

Figure 2.16. Create a test table "A".

[View full size image]

4.
Next, log into the database using Query Analyzer with the TestPerson credentials. Issue the same Create Table DDL and refresh the Tables node in Enterprise Manager. Notice that this creates another Table A, but this time with a prefix of TestPerson, as shown in Figure 2.17.

Figure 2.17. Displaying the two "A" tables showing the owner assignment.

So this is nothing new, right? In SQL 2000, objects such as Tables, Functions, Views, SPs, etc. take a prefix of their owner. A SQL 2000 object fully qualified is a four-part name:

<Server>.<Database>.<Owner>.<Object>

When using SQL 2000, programmers don't usually use the fully qualified names in T-SQL. Actually, this is not necessarily all that bad because I've seen databases where novice programmers thought they were smart by fully qualifying the names in T-SQL stored procedures that deleted entries from tables that contained cash income. Being ever so smart, they realized that it's not considered good form to work on a live database (and I agree). To prevent damage to the production data, they created a demo database on the same machine and started executing the stored procedures to clear entries. Unfortunately, while they thought that they were clearing entries from the backup database, they were actually removing data from the live database because of the fully qualified addressing.

So you sit there thinking to yourself, "That knucklehead. He should just work with the last part of the object name." Okay, so using the Enterprise Manager (or your tool of choice) put some data into the two tables dbo.A and TestPerson.A. Make it easy on yourself and put "dbo" in TestCol of the dbo.A table and "TestPerson" in the TestPerson.A table. Okay, so let's try our luck out with a SELECT query. Take a look at the examples in Figure 2.18 and Figure 2.19.

Figure 2.18. Adding rows and executing a T-SQL SELECT using default and explicit addressing while logged in as "dbo".

Figure 2.19. Adding rows and executing a T-SQL SELECT using default and explicit addressing while logged in as "TestPerson".

[View full size image]

The TRUNCATE TABLE operator in SQL Server drops all rows from a table. It's dramatically faster than executing a DELETE or dropping the table and all of its dependencies and rebuilding it.

Actually, the way SQL 2000 works, if the Owner portion of the object name is not specified, SQL Server tests for an object that belongs to the connected user with the current user name before searching for an object of that name owned by the dbo. Based on this behavior, from a performance point of view (albeit, marginally better), all objects in T-SQL statements should specify the owner as well as the name. Otherwise, some redundant mapping takes place most of the time when an object is referenced without a specified owner.

In many development shops, the DBA/sysadmin (the dbo) doesn't want to give blanket rights to developers. They don't want just anyone creating objects owned by dboespecially during development. This leads to the situation where it's expedient for objects to be created that are owned by non-dbo creators.

Suppose our TestPerson has been fired (his job was outsourced to WhoKnowsWhereIstand) and the DBA/dbo finds that they now wish to delete the TestPerson account from the database. Go ahead and try it. If a user owns objects in the database the owner can't be droppedat least not easily. The problem is that in SQL 2000 and earlier versions, the formal "owner" and "schema" were effectively one and the same.

How SQL Server 2005 Addresses the Ownership ChainIssue

In SQL 2005, schema has been separated from owner. From now on, the fully qualified name of an object is:

<server>.<database>.<schema>.<object>

Okay, so you perhaps wonder why this is such a big dealafter all it's not too onerous (groan) to change the owner on objects to another user and then drop the original user. Yes, this is true, but the problem is a little trickier. That's because you could potentially have client applications that are using objects qualified by the owner. This is especially true if those clients applications connect via a trusted connection and the object was not owned by the calling user or the dbo.

Most SQL Server DBAs and developers don't really create logins or user accounts based on people's namesI hope you don't, either. We do create accounts for business functions such as "PayrollClerk2" or "PointyHairedBoss". We also create SQL Server roles for these functions and assign rights to the roles. This (generally) eliminates the need to change the database or applications when someone leaves the company or gets tossed out on their butt.

Think of it this waysuppose I have an application that executes T-SQL such as:

SELECT TestCol FROM TestPerson.A

If I manage to change table A's owner to AnotherTestPerson.A and subsequently delete the TestPerson user from the database, the application is now broken until the application is re-written to refer to table A as:

SELECT * FROM AnotherTestPerson.A

This can be an expensive process if one has to touch each and every program, component, function, stored procedure, and trigger that refers to these hard-coded objects. There has to be a better wayand there is: In SQL Server 2005, objects are effectively "contained in" (not owned by) schemas, and schemas themselves can be owned by different users. This makes authoring manageable ownership in T-SQL much easier because the ownership of a schema can be changed without affecting applications. I'm trying to cajole Peter into writing an EBook on ownership that walks through this in more detail. Perhaps he'll relent before Microsoft changes it all again.

Understanding Logins, Usernames, and Schemas

As I just said, SQL Server supports both logins and usernames to implement authentication and authorization. Logins can be either Windows login accounts such as MyDomainName\billva or SQL Server logins such as "SA". By default, only Windows authentication credentials are supported, but if you enable mixed-mode security on your server instance, you can use both types of logins for authentication.

When you (or the DBA) grant a specific login name access to a database, that login is linked with a username in the user database. As a result, a corresponding security link (an SID) is recorded in the master database. This means that each username in the user database maps to a login in the master database. When adding login accounts, one should specify a default database, as shown in the Management Studio's "create new login" dialog, shown in Figure 2.20. However, before you can do this, you must create the Windows login on either the local development system or (more typically) the Windows domain. In the example in Figure 2.20, I created a local machine account (OCTO4CTP\DevLead) and used Management Studio to register this login with SQL Server.

Figure 2.20. Creating a new user and assigning the default database.

[View full size image]

Once the login has been added to the list of recognized login accounts, Management Studio adds it to the sysxlogins table in the master database and indicates the default database this login is to use; the next step is to specify what database(s) this login is permitted to access.

To choose one or more databases for this login, click on the Database Access icon (upper-left corner of Figure 2.20). This launches a new dialog and permits you to choose one or more databases. After you check a database in the dialog, choose one or more appropriate roles for the username on the selected database.

Repeat this role-selection process for each database the username is to access. Yes, you can also drill down into the database|security|users tree and set the username rights there, as shown in Figure 2.21.

Figure 2.21. Choosing a database and rights for that database.

[View full size image]

Understanding SQL Server Roles

SQL Server supports nine fixed database roles, as shown in Figure 2.21. Each of these roles defines the rights the specified username is granted when accessing the database. The basic roles are:

  • db_datareader: This role permits username accounts to read any data in any user table in the database.

  • db_datawriter: This role permits username accounts to add, change, or delete data in any user table in the database.

  • db_owner: This role permits the username account to perform all configuration and maintenance activity on the database. This means db_owner role members can change the database schemaadd, change, and delete objects within the database.

  • public: All usernames are members of the public role. You can grant or deny rights to public on a case-by-case basis.

There are six other roles supported, but these are generally of more interest to DBAs and more advanced users. Check out books online for more information on fixed database roles, if you're curious. If you don't assign a role to the database username, it inherits the rights granted to the public database role.

As you can see, these roles are not particularly granularthere are many situations where you're going to want to grant access to specific tables (or even columns within tables) to specific usernames and not to others. This is a fairly involved process that involves considerable planning. Enterprise Studio has a new set of dialogs that permit you to set very specific rights on any object in the database. It's beyond the scope of this book to get into the details of this process.

Understanding Object "Ownership"

By default, the owner of any object is the username that created the object or the schema to which that user belongs. Generally, your DBA won't grant "create object" (db_owner) rights to all usernamesonly to a select few. Keep in mind that logins don't own or have rights to do anything except authenticate access to the databaseit's the username associated with the login that's assigned rights to objects in the database. No, a database is not considered an "object" that can be owned, so databases are "owned" by loginsnot usernames. Only usernames own objects within a database. An object is one of the components in the database, including tables, indexes, triggers, views, keys, constraints, defaults, rules, user-defined types, or stored procedures.

Handling Special Accounts

There are a couple of "special" accounts that need mentioning at this point. By default, SQL Server exposes a "System Administrator" (SA) login accountit's never a username. The other special account is the "database owner" (DBO), a username that maps to a login that's the database owner. At one point in time, most books, documentation, and training sessions used the SA account to log on to SQL Server in their examples. While I still see this practice from time to time, it's generally frowned upon. The problem with using SA as the login account is that production applications should (generally) not use this account unless they are performing a special administrative function. Yes, if you log on to SQL Server using the SA login, your application is granted rights to do anything on the server instanceon any system database or user database, regardless of whom or how it was created. That's because the SA account holds the "skeleton master key" to everything SQL Server instance stores. It's critically important that the SA account password be protected at all costs. Actually, the SA account and its password are hidden during a default installation. You'll need to request "mixed-mode" security to expose it and to set its password. Again, I don't recommend this approach, but if it's necessary, I strongly recommend using a strong SA password and creating a mechanism to change it on an irregular[7] schedule.

[7] I say "irregular" schedule to help throw off those who depend on regularly scheduled security changes to make it easier to crack into a system.

IMHO

I rarely recommend using the SA account to develop production applications, and only rarely use it to perform system maintenanceexcept when absolutely necessary.

There are other issues you'll (eventually) encounter when using SA as the development or production login. First, you're masking permissions issues that should be dealt with early in the database design phase. You'll find it tough to rearchitect an application that does not have a robust security plan woven into the design. In addition, at one point in time or another, you're going to have someone do a security review on your implementation. If they discover you've used the SA accounts in production, you had better have your resumé up-to-date.

All of that said, consider that the new SQL Express "User Instance" paradigm encourages developers to use the user account (with Windows Authentication SSPI security) to get SA access to the local "user-owned" instance of SQL Server Express. This way, the user account is granted full SA rightsthe user account "owns" the instance and all it contains. This also means the application can do anything to the SQL Express user instance, including dropping the database, adding new databases, or anything else short of bringing global warming to an end.

Understanding SQL Server Connection Security

By default, SQL Server is configured to use "Integrated" security, as described in Chapter 1. This configuration is also called "Trusted", "SSPI", or "Windows Domain-managed" security. This means that when a connection is opened, SQL Server expects to be passed the Windows domain credentials of the login account that's currently logged into the Windows workstation. SQL Server looks up this value in its list of known login accounts and assumes the password is correct as it was validated by Windows. This login name can be a person (in which case, the person's Windows credentials are passed to SQL Server for authentication) or an application like IIS hosting an ASP or ASP.NET application (in which case, special "ASPNET" [in IIS 5.0] or "IIS_WPG" [in IIS 6.0] account credentials are authenticated).

Windows domain administrators are automatically granted SA rights on SQL Server. This is another reason not to use administrative accounts to develop or deploy applications.

Windows domain administrators are automatically granted SA rights on SQL Server, so if you must perform operations restricted to the SA, you need to log on to Windows as Administrator or as a member of the Administrators group assuming your IT people have given you these rights (I don't encourage them to do so). If the user logging on to SQL Server is not a member of the domain Administrators group, they need a login account of some kind on SQL Server. Yes, you can define your own custom Roles on SQL Server that contain one or more logins or domain groups, as discussed earlier. This way, you can create a functional group of users that are recognized by SQL Server and granted specific rights to the resources they need on the server. You can also include a Windows group into a custom role to permit the domain Active Directory help manage which Windows logins are authenticated to access the database(s) and the objects they contain.

A new "schema"-based security model in SQL Server 2005 separates users from objects, providing more granular control of data access objects.

When you connect with ADO.NET using mixed mode (SQL Server) security, you must include a hard-coded login and password credentials that match a login and password registered on the SQL Server instance. In this case, the parameters passed to the data access interface in your code must include this login name (ADO.NET calls it a "LoginID") and passwordexposed for all of your development team to see[8]. This approach won't work at all unless mixed-mode security is enabled on the SQL Server instance you're addressing.

[8] I show you ways to hide these credentials in Chapter 9, as well as ways to ensure that exposed credentials can do no harm.

To set up this and every user login and password or set up groups of several related logins, you'll need to log on to SQL Server as a member of the domain administrators group (or as SA, if your instance supports mixed-mode security). While I discuss the basic details of setting up these accounts later in this chapter, this approach is generally not recommended. However, if you choose to use mixed-mode security accounts, you'll want to take steps to protect the credentials.

One technique I recommend is to create an application-specific login instead of requiring a user to remember a unique login and password. This is especially useful when you don't really know your users, as when building anonymous logon ASP applications or XML Web Services. This is relatively safe, as an application login is configured, so it can execute only specific stored procedures, so little harm can be done if the credentials leak out. In any case, an individual login or role containing the login must be set up to permit access to whatever user database and objects are appropriate.

No, as I've said before, I don't usually create login accounts for "people," but for their function. For example, I'll create a Login for "AccountPayableClerk1" and let the accounts payable application log into SQL Server with these credentials. This particular login is added as a username in the Accounting database and granted permission to (just) the database objects needed for the applicationand no others. I rarely grant specific usernames access to base tables in the databaseit's simply too dangerous. I usually build stored procedures on the server to perform all of the required queries and updatesbut I'll get to that later.

So in case you missed it, you can build applications that don't require users to enter SQL Server login credentials, as the application can use the credentials assigned to the user when they logged into Windows. On the other hand, the application can use hard-coded credentials or, as in the case of an ASP application, use the credentials assigned to IISthe "ASPNET" or "IIS_WPG" account. Yes, it's also possible through "impersonation" to have Windows credentials to be passed through IIS to SQL Server, but this works only in certain configurations and is more expensive to implement. In Chapter 9, I discuss how to open connections using each of these connection strategies.

Creating Security Personas

At this point, let's try to implement some of the theory and suggestions I've covered earlier in this chapter. So, before you start wading into SQL Server application development, you should consider creating and configuring security personas for at least three types of accounts:

1. | The SQL Server System Administrator. This persona is responsible for dealing with administrative functions on SQL Server and should not be used by developers to create, tune, or test applications.
2. | One or more "developer" personas. These accounts might be created with quite a few rights as their role, as a developer dictates. Depending on the size of the shop and the sophistication of the security scheme, the developer personas might be restricted from accessing (seeing, modifying, and tampering with) sensitive tables, views, or stored procedures.
3. | One or more "client" personas. These accounts are created with very few rights, generally just enough to perform the tasks executed by the applicationthat is, rights to execute specific stored procedures, access specific views, and (rarely) read specific tables.

As you create your database and add tables, procedures, and rights to these objects, you'll use the administrator's account. Your limited-rights developer persona is used by your development team to create applications that use the aforementioned SQL Server objects. The client account is used when testing your application. In each case, you can log on to Windows using the appropriate account or use the "run-as" approach that executes the Visual Studio IDE or your test executable as a different identity.


Managing SQL Server Connections

To access the data or manage the objects in SQL Server, your application or the tools you use have to open a link to the networked or local SQL Server instance's serviceassuming you've already installed the server. This link is called a "connection," and it's typically implemented behind the scenes by exposing specific TCP and UDP ports (that you can specify by number, if necessary).

One of the most troublesome aspects of working with SQL Server can be establishing and managing one or more connectionsthat's why I devote all of Chapter 9 to connecting and connection issues. Why? Well, SQL Server protects itself and your data by preventing unauthorized users from connecting, so it's designed to guard against casual access (so to speak). These security constraints on Windows, the network, and SQL Server (in several places) prevent you and everyone else from gaining unfettered access to the server and the data unless you have proper authenticated credentials.

Until SQL Server 2005, you couldn't just use a program to point to a SQL Server .MDF file and open it. You still can't open a .MDF file and make much sense out of itonly SQL Server can open and manage its binary data files. However, all it takes is physical access to the .MDF to compromise its datasimply transport it to another system containing a friendly SQL Server instanceand....

Before I go on, let's make a few terms clear. When an application (any code anywhere) opens a connection to SQL Server (or any DBMS data source), it's considered to be a "client." This application can be embedded in a Windows Forms application, in a middle-tier XML Web Service, in an ASP application, or on a hand-held PC connected to a milking machine. "Client/server" applications are considered to be those programs (like Windows Forms applications) that open connections and provide interactive user interfaces, and are not browser-based. That is, the application runs on an independent computer separate from the system that hosts the DBMS engine (like SQL Server). A "server" is the system that hosts SQL Server or other shared DBMS. Yes, a server can share files and it's then dubbed a "file server." SQL Server gets its name from the fact that it provides Structured Query Language (SQL) "services" to the network.

Understanding the Data Access Interface Role

In Chapter 1, I showed you how Microsoft has exposed a number of data access interfaces. These include DB-Library, ODBC, DAO, RDO, OLE DB, ADO, and now ADO.NET. This, of course assumes that Microsoft has not created another TLA since this book hit the streets. For good or bad, each of these interfaces requires a separate (and distinct) "stack" of DLLs to implement their functionality. This has meant developers must install the correct stack for the application they're deployingone that matches the interfaces they used when the application was developed and tested. As I have said, when forcing the application to "upgrade" an existing stack on the target system, developers saw an alarming number of incidents where existing code misbehaved (to be kind). Most notorious of these is the MDAC stack, which is used to support OLE DB, ODBC, DAO, and ADO (classic). Over the years, its constituent parts have changed, but it has (at one time or another) contained the JET engine, OLE DB drivers for a variety of backends, ODBC bridge providers, and several other sundry DLLs.

Understanding the Protocols

For the most part, you needn't be concerned about which protocol is being used to transport TDS packets to SQL Server. If you do care, ADO.NET and the data access interfaces support four different protocols. Each has its "best use" scenario and limitations:

  • TCP/IP: The "Transmission Control Protocol/Internet Protocol" is the standard (open source) protocol used to move data over the Internet. It addresses source and destination locations using a TCP/IP address and can be routed anywhereexcept parts of rural Leavenworth County, Kansas. This protocol is the most flexible but not the most efficient.

  • Named Pipes: This protocol is used to move data over local area networks in a Windows domain. It requires that the host server (hosting SQL Server) be capable of creating a named pipe interfaceonly NT-class operating systems are so capable. This means Windows 9x or ME cannot expose SQL Server instances using named pipesyou'll have to use NT, XP, or Windows 2003 Server (or better), or TCP/IP protocol. Peter tells me that Named Pipes is evil, as it's not really a routable protocolwho am I to disagree?

  • Shared Memory: This protocol is used to link SQL Server within client memory on the same system. It bypasses the "network" layers so it's the fastest due to the lower overhead. By default, this is the only protocol SQL Server 2005 instances support. All other protocols have to be specifically enabled.

  • VIA: "Virtual Interface Adaptor" protocol. Used on more recent NT (and later) OS builds, this protocol is designed to work with specific VIA LAN hardware[9].

    [9] VI Protocol: See www.windowsitpro.com/Article/ArticleID/16097/16097.html

As I discuss coding the ADO.NET connection string in Chapter 9, I'll show how to choose the protocol that makes most sense.

Opening a Connection

To open a connection to SQL Server, ADO.NET is used to capture a number of parameters and pass these on to the .NET provider. These include the TCP/IP location or DNS name of the SQL Server, the instance being addressed, appropriate security credentials, and a number of optional configuration settings. These parameters are placed in the ADO.NET SqlConnection object property called the ConnectionString and referenced when you execute the ADO.NET Connection.Open method. That's it. When creating a connection from one of the tools such as SQL Server Management Studio or Express Manager, you provide the same parameters via an interactive dialog (as shown in Figure 2.22). The mechanism behind the scenes is the same.

Figure 2.22. Opening a connection to a SQL Server instance using SQL Server Management Studio.

There are a dozen or more other optional parameters (expressed as key ; value pairs) that help your application point to a specific user database on the server; name your application; specify the I/O port, the protocol, and the data cache size; or change how the data access layers manage the connection and the connection pool for you. I'll discuss how to manage these options in ADO.NET in Chapter 9.

When SQL Server is asked to open a connection, the client layers are forced to load (or locate) specific DLLs, network libraries, and other support structures; locate the server (machine) using the Active Directory and the DNS; and address a specific SQL Server instance on the network based on the Windows domain name, (or an IP address) and TCP/UDP ports, and the proddings of a good acupuncturist. Once the server and instance are located (of course, each server machine can support multiple SQL Server instances), it's queried to see if it can support another connection (based on license and other restrictions). If a connection is available, it's assigned to the data access interface requesting the connection and an agent (worker thread) is started on the host SQL Server instance to manage any queries requested by the application at the other end of the connection. This can take quite some timeespecially the first time, as DLLs and other overhead tasks must be initialized on first use. After that, subsequent connections can be established more quickly, but it's still a relatively expensive process.

When trying to open a connection for the first time, if you find it takes too long to open the server using a DNS (machine) name, you might be having problems with a slow (or non-existent) DNS server. Try putting the machine name along with its hard-coded TCP/IP address in the hosts file.

The next time your application has to open a connection to the same server, only some of these steps need to be repeated. However, each time you open a connection, the supplied credentials are revalidatedeven if they have been validated a thousand times in the last five minutes. This takes timeeven if you're fetching a connection from the connection pool. However, many of the other steps have already been done, so connecting for the second time can be far fasterassuming the pool hasn't already been exhausted. See Chapter 9 for more details on how the connection pool worksor is supposed to work.

Unfortunately, SQL Server is not particularly informative when it complains that it can't connect. Part of the problem is the TDS packets that the device drivers and providers return, and part of the problem is how the connection process is handled behind the scenes. For whatever reason, when you don't get connected, you're likely to get a less-than-useful message that says that something went wrong. Fortunately, the .NET Framework and SQL Server 2005 have improved the quality and usefulness of these messages. While they might be hard to decipher, they usually contain several clues as to why you didn't get connected. For example, when trying to create a new SQL Server registration in SQL Server Management Studio, the dialog shown in Figure 2.23 was exposed. This more-or-less generic message says that the problem could be the server name, login credentials, or database chosen. To get more information, I clicked on "Show advanced information".

Figure 2.23. Initial exception message in SQL Server Management Studio register server dialog.

[View full size image]

This exposes a dialog that provides a lot more (albeit complex) information about the exception. Here, you can see which interface (TCP/IP, named pipes, shared memory or via) was being used (named pipes, in this case) to open the connection and (roughly) what when wrongas shown in Figure 2.24. I can see that the network path was not found. This means the network could not resolve the name of the server.

Figure 2.24. The advanced information dialog provides details about the connection exception.

[View full size image]

Since I tried to connect using a server name, I have to figure out why the named server could not be found. The first thing I usually do is "ping" the server by name using the Ping utility from a command prompt, as shown in Figure 2.25. In this case, I can see that the server name resolved to an IP address. This address was found in either the hosts file or the directory name service (DNS). However, ping could not get a response from the server. This means the server itself or the network link to the server is down.

Figure 2.25. Using the ping command-line utility to test for the presence of a server on the network.

At this point, you need to determine:

  • Is the server name correctly spelled? Server names are not case-sensitive, but you have to spell the name correctly.

  • If an IP address is used to address the server, does the address include the correct port number using the syntax xxx.xxx.xxx.xxx.:pppp, where pppp is the port number?

  • If you're using a server name with no instance name, is SQL Server installed on the server as the default instance?

  • If you're using either "(local)" or "." to reference the local server, is SQL Server installed as the default instance on the local system? This syntax won't work with named instances unless you include the instance name using the ".\instancename" syntax.

  • If the SQL Server is installed as a named instance, is the name included along with the server name using the server\instance syntax?

  • Are you connecting to a local instance or a network instance?

  • Is the network link to the server up? If you're using a VPN connection to expose the remote server, is it enabled?

  • Does the ISA server on the server-end of the VPN permit dialup access to the server?

  • Is the server configured to expose its I/O ports (usually 1433, but not always)?

  • Is the server running the SQL Browser service so that it can be seen on the LAN?

  • Assuming the server is found on the network with ping, does the server recognize the credentials provided?

  • Is there a login account that matches the credentials? This might be a role that contains Windows domain groups or individual SQL Server logins.

  • If the credentials are recognized, do they have rights to the default database as assigned by the system administrator when the Login account was set up?

In Chapter 9, I'll add to this list as I discuss connecting to SQL Server from ADO.NET.

Debugging a Connection

I've provided a flowchart (shown in Figure 2.26) to help you walk through the process of opening a connection. As you can see, there are a quite a few issues to keep in mind. Notice how SQL Server uses (or demands use of) specific data access transport providers such as TCP/IP or named pipes. I also show what to do at each step to help build the ConnectionString or to diagnose what went wrong. While this chart won't help find all of the possible issues you'll encounter over your lifetime as a developer, I expect it to help get connected in the majority of the cases you'll have to deal with.

Figure 2.26. Flowchart showing how to debug a connection.

[View full size image]

Using Connection Pooling

In an attempt to make the process of opening connections less expensive (after the first connection), many years ago, Microsoft implemented a technology called "Connection Pooling". It's still alive and kicking in the 2.0 version of the .NET Framework. ADO.NET 2.0 permits even tighter control over this automated connection management interface. Basically, the connection pool is a software mechanism that runs on the client (it's not managed by SQL Server) that's implemented by the data access provider. That means the ODBC, OLE DB, and .NET drivers and providers each implement connection pooling on their owneach has its own properties and setup options. When I get to the ADO.NET SqlConnection object in Chapter 9, I'll show that the SqlClient .NET data provider has a wealth of properties, counters, and methods to monitor, control, and even disable the connection pool.

Each process gets its own connection pool. In the case of Windows applications, that means each application gets its own pooleven if 10 instances of the same application are running on the same system. In ASP/ASP.NET applications, each application domain gets its own pool. This means multiple instances of ASP pages share a common poolas long as the ConnectionString remains the same[10]. When an application requests a new connection (using the Open method), the following operations take place:

[10] There are a number of other factors that determine when additional pools are created; these are discussed in Chapter 9.

  • The connection pooling mechanism searches the pool for a matching connection string (that contains the credentials and connection settings).

  • If a match is not found, a new connection is established. This process involves authenticating the credentials, locating the server, passing the credentials to the server for revalidation, locating the default database, and authenticating access by the given login.

  • The server creates an "agent" to handle the connection.

  • The connection pooling mechanism passes back a pointer to the new connection to data provider.

When another connection is needed, the process is repeated. If the existing connection is still in use, the process is repeated and another connection is added to the pool. When your code closes a connection, the connection pooling mechanism marks the pooled connection as "dormant" so it's available when your application attempts to open the connection again. After four to eight minutes[11], the connection is automatically dropped from the pool and the "real" connection to SQL Server is closed. When this happens, the server releases all of the resources owned by the connection. Note that the connection pooling mechanism does not close the connection to the server when ADO.NET executes the Close methodit's simply returned to a pool.

[11] No, you can't programmatically alter this dormant to closed timeout period.

When your application attempts to establish an identical connection using the ADO.NET Open method (using an identical connection string), it's simply passed to the requesting application, thus saving the cost of establishing a new connection. If your connection string changes (as when you change the user credentials, default database, or any other setting), you'll get a new connection pool to manage other connections using the same connection string.

Before the application is passed the "used" connection, the connection pooling mechanism revalidates the credentials. In addition, when a reopen is requested, the connection pooling mechanism resets the connection (during the first operation on the connection), so any resources held by the previous owner are released, so your application can inherit a clean connection each timethis also takes time. It turns out that since SQL Server 2000, the reset request is tied to first request to use the connection and does not take a separate round-trip. The problem with this approach is that the resources can be held on the server for several minutes (48), which is a very long time. Basically, this means reconnecting is not freein some (many) cases, it will be faster to leave the connection open for the lifetime of the application. That's what I often recommend for Windows Forms/Smart Client applications.

Managing Multiple Active Resultsets (MARS)

SQL Server can support hundreds to thousands of connections. In earlier versions, each connection could perform only one operation at a time. That is, you could execute SQL that returned a set of rows, but you could not use the connection to do something else until all the rows had been fetched and the connection was freed for reuse. It's like using an old-fashioned two-way radio. You could talk or listen, but not both at the same time. This is called a half-duplex connection. SQL Server 2005 supports new technology called Multiple Active Result Sets (MARS). This permits you to use the same connection to perform one, two, or more operations at the same timea full-duplex connection. Whether or not this makes sense (since individual connections have other benefits) is TBD. Microsoft added this functionality to help developers create applications that can more easily work against either SQL Server or Oracle (which has supported multiple operations per connection for some time). As you'll discover after having read Chapter 9, I'm no fan of MARSor the third moon of Jupiter.

Choosing a Connection Strategy

When working with Windows forms applications, you might consider opening a connection and leaving it open for the lifetime of the application. This is especially useful, as when using this approach, you'll be able to manage server state, including building and reusing temporary resultsets or global variables as well as server-side cursors (that you'll have to create manually). The downside to this approach (in some folks' mind) is that each connection consumes resources (RAM and CPU time) on the server. More connections mean less RAM available for the cache that helps SQL Server performanceeach connection costs about 40K. However, this problem can be addressed by adding more RAM to the server. This approach does have its limitsI expect an average server would be hard pressed to support more than several hundred users with an average application. Since the vast majority of SQL Server systems support hundreds instead of thousands of users, this should not be a problem. As I discussed a little earlier, it's not free to close and reopen connections, as the connection pooling mechanism still has to revalidate and reset the connection on each open.

When working with other architectures such as IIS/ASP applications, you can't get away with leaving the connection openyour ASP page does not live that long and applications that hold open connections are impractical (but not impossible) to write and hard to scale. Almost universally, these applications launch a selected executable page, open a connection, execute a query, close the connection, and wait to be reactivated to post the changes to the server. In this case, connections are opened and closed like the turnstiles on a subway platformoften thousands of times a minute. Ordinarily, this would slow the server down quite a bit, as the process of opening a connection is fairly complex and expensive, but the connection pooling mechanism reduces the overhead of this connection thrashing to a minimum. Because the connection is not left open, ASP applications can't depend on SQL Server server-side statethey can't expect to build and reuse temporary tables or cursors as they can with Windows Forms applications. However, this type of application scales up far higher than a typical Windows Forms application. They also expose the database to anyone on the planet, so they need especially tight security.

SQL Server speaks one and only one language, "Tabular Data Stream" (TDS). This is a proprietary protocol that encodes the conversations between all clients and the server. Each version of SQL Server uses a subtly different version of TDS, but the server is capable of recognizing older versions and executing their requests. At one time, Sybase's version of SQL Server used TDS, but it was abandoned many years ago, so you won't be able to use TDS to communicate with current versions of Sybase SQL Server. TDS dictates several aspects of the data that's sent and received from SQL Server. For example, when the data access layer (like ADO) needs to retrieve the results of a query, it has to wait until all rowsets (as generated from SELECT statements) are returned before the RETURN value or OUTPUT parameters are returned. TDS also determines how exceptions are handled, how MARS works (or doesn't), and many other factors that are really outside of the control of ADO.NET (or any other data access interface to SQL Server).

Managing Server State

As your applications get more sophisticated, you'll find that you might be able to help performance or tune the way your queries are executed by managing server state yourself. Server state is simply the settings and objects owned by the connection on the server. For example, one of the easiest changes you can make is to set or change the initial catalog/default database. When working with ADO.NET (or any data access interface), you can set this value when you connect by adding "Database=<your default database name>" to the ConnectionString property. Yes, there are other settings you can make using the ConnectionString. You can also create other objects on the server that are persisted as long as your connection remains open. These include temporary tables, server-side cursors, pessimistic locks, and various settings used to control the behavior of the T-SQL compiler and SQL Server itself. These settings and temporary objects can play a major role in how SQL Server handles dates, locking, query execution, transactions, arithmetic issues, and much more.

Some of the settings you can change have an impact on all connectionsnot just yours. This means you should not make changes to these settings until you fully understand their impact. Generally, when a stored procedure or trigger makes a change to a setting (by executing the SET statement), the setting is rolled back after the procedure is done. This is also the case when the query is executed with sp_executesql (as ADO.NET often does). There are a couple of exceptions: SET ANSI_NULLS and SET QUOTED_IDENTIFIERthese settings are persisted and cannot be changed in a stored procedure. In some cases, these settings impact not just your database, but all databases on the server. See "SET (described)" in BOL[12] for more information.

[12] BOL: SQL Server documentation (Books Online).

The connection pooling mechanism rolls back any connection-owned settings and objects (like temporary tables and cursors) when the connection is reset on reuse. Actually, cursors are closed and their resources are released when the client executes the Close method. Why is this important? Well, if you want to create your own temporary (#Temp) tables or server-side cursors (ADO.NET won't do it for you), you'll need to remember that these objects are owned by the connection and live only as long as the connection. In addition, these objects consume resources on the serverand that limits scalability. If you need to support fewer users than a public web site selling cheap drugs, SQL Server can easily handle a couple hundred server-side cursors and #Temp tablesassuming you don't overload them with too many rows.


Using the SQL Query Tools

As you and I travel though this and subsequent chapters, you'll notice that I illustrate many of my points using T-SQL code samplesI've already shown you a few of these. Since you'll probably want to try these out yourself, it's time to show you how these samples are edited and tested using Visual Studio and SQL Server Management Studio. If you've worked with SQL Server 2000 (or earlier versions), you're accustomed to using the Query Analyzer (the GUI query editor and tester) or the command-line tools such as ISQL or OSQL to generate, edit, and execute your T-SQL data description language (DDL)[13] queries. You might also have used Visual Studio's "Server Explorer" to perform these tasks. If this is the case, you're well ahead of the game.

[13] DDL is the language used to define databases, tables, and all of the other database objects.

In SQL Server 2005, all (or most) of these familiar tools have been rolled into a single application: SQL Server Management Studiothis is the tool we'll be using to illustrate how SQL Server manages the DDL and Data Manipulation Language (DML) tasks needed to create and manage databases and the objects they contain.

Creating a New Query

It's really pretty simple to create a new T-SQL query. Just follow these steps:

1. | Start SQL Server Management Studio. It's in Start | All Programs | SQL Server 2005.
2. | Open a connection to your SQL Server instance using the connection for the selected server. The first time you do this, you'll be asked to register the target server instance.
3. | Click on New Query. This opens a query editor window. You can also drill into Tables, Views, Stored Procedures or other objects; right-click; and choose to have Management Studio generate a template to modify an existing object or create a new object.
4. | Enter or modify the T-SQL. Don't worry about case unless your SQL Server instance is installed as "case-sensitive".
5. | Choose the database against which the query is to execute from the dropdown.
6. | Click on the "check mark" to test the syntax or on "Execute" to run the query. Note that if you select one or more lines before clicking on the test syntax check mark or Execute buttons, only the selected TSQL is tested or executed.
7. | View the results in the lower "Results" tabbed window and any messages returned in the "Messages" tab.

Figure 2.27 shows you what you should see at this point. I've created a T-SQL query and highlighted the areas you're likely to use most often. Notice that I executed the query, so the results window shows the rows returned.

Figure 2.27. SQL Server Management Studio's TSQL query editor.

[View full size image]

Keep in mind that SQL Server Management Studio does not show a "live" view of the objects in the Object Explorer. That is, the list itself won't show new tables, columns, stored procedures, or anything else after they are created. You'll find yourself right-clicking the object header (like "Tables") and selecting "Refresh" to get SQL Server Management Studio to requery the system catalogs for an updated list. This also applies to data queries. The tool lets you open a window to inspect the contents of the database and even edit the data. Simply choose a table, right-click, and choose "Modify". This runs a SELECT * query against the table and displays the rowset in a new window. If the data changes, this window will not show those changesunless you're making them interactively in the window. To get a fresh copy of the data, click the "Execute" button.

I discuss how to use other features of SQL Server Management Studio to view and tune the query plan later in this chapter.


Creating SELECT Queries

Once a connection is established, SQL Server launches a process (identified by a process ID or PID) that permits the client application (through ADO.NET via TDS) to submit Structured Query Language (SQL) queriesthus the name "SQL Server". The TDS protocol does not care what kind of string ADO.NET sends as a "query"it's simply a transport mechanism. Sure, you could put a letter to your Aunt Mildred in the T-SQL string, if you're so inclined. When the server processes the TDS packet, it "compiles" the string as a T-SQL query. This process involves a simple syntax check followed by additional steps to ensure the objects referenced in the SQL actually exist and the connected user has sufficient rights to the objects. In other words, SQL Server checks to see if the database, tables, stored procedures, or other objects or elements you reference actually exist in the master or user database catalogs of objects and that the current user (the one that was permitted to open the connection) has been granted permission to "see" the objects referenced.

If the T-SQL syntax check fails (and it should fail if you sent Aunt Millie's letter as a query, unless, of course, she expects a SQL queryin which case she needs counseling), a TDS packet is returned with details about the problems, and ADO.NET throws an exception with the details in the Exception Message property. These details can be pretty skimpy, as only the first layer of errors is returned. If you've worked with Query Analyzer, you've seen these syntax errors (as shown in Figure 2.28).

Figure 2.28. SQL Server checks your SQL for syntax errors before executing it.

Frankly, I've found that SQL Server Management Studio provides a better editor than Visual Studio for authoring SQL statementsthe error handling is more informative and complete. However, even SQL Server Management Studio does not always get the line numbers correct. That is, it sometimes gets confused by comments and ancillary commands. I'll show you how to edit SQL in both SQL Server Management Studio and Visual Studio in later chapters.

If the syntax of your query is acceptable, the next phase of the query compile can also return exceptions if the objects your SQL references (databases, tables, columns, views, stored procedures, and so forth) don't exist or the connected user doesn't have permission to access them. These exceptions are passed back to ADO.NET via TDS, and your application (or Visual Studio) throws an exception. If the object references are valid, the query optimizer is executed to determine the optimal query plan, given the submitted TSQL.

The SELECT statement is the SQL[14] operator you use to choose acceptable rows from a relational database based on criteria you include in the statement. While I don't really show you how to create SELECT statements in this book (there simply isn't room), consider that a simple SELECT statement (shown in Figure 2.29) specifies the table columns to return (Au_ID and Author) from a specified table (Authors) and returns a set of rows containing data from the specified table columns (if the condition specified in the WHERE clause is metin this case, rows where the Year_Born column value is between 1945 and 1947). The row(s) that results from any SQL query is called a "product" in SQL terms (in case you're asked this question on Jeopardy!).

[14] T-SQL supports ANSI SQL statements and many other statements not supported in the ANSI standard.

Figure 2.29. A (very) simple SELECT statement.

The SELECT statement usually specifies (and possibly names) the data table columns to be included in the rows to be returned. One option you'll see used very often when coding a SELECT statement is use of the "*" operator to indicate that all columns should be returned, as shown in Figure 2.30.

Figure 2.30. Using the SELECT * option to return all columns

Yes, this seems easy, but it's not a good idea to use unless you're writing table-management utilities. To make your applications perform better and provide explicit column referencing and fewer problems debugging later, I recommend that you explicitly specify each column you need to reference. One of the problems is that when you use SELECT *, you never know what is going to be returned by the query sometime in the future as the table schema matures. Someone could add a TEXT or IMAGE (what we call binary large object, or BLOB) column to the table, and your query could unknowingly pull back far more data than you need. You'll also discover that the Visual Studio tools now convert many SELECT * queries to return the specific columns by name.

In any case, you must make sure that the column names you specify are unambiguous. That is, the query compiler (and your code) must know how to address the data from the source tables using unique column names. Because SQL Server won't let you create two columns with the same name in a single table, this is not an issue until you pull data from more than one column at a time using the JOIN syntax (which is discussed later in this chapter).

To make the job of the compiler easier, each column in the database is addressed by four namesthe database name, the table owner's (or schema) name, the table name, and the column name. For example, to address the Price column in the Parts table, which is owned by Fred in the Pieces database, you would code: Pieces.Fred.Parts.Price in the SELECT statement. Sure, one can shorthand most of these names. First, you can usually leave off the database name because one can (should) set a default database (or initial catalog) when opening a connection. A connection points to a default database, but you can change it in code with ADO or add a USE <database> to your script. If you want to JOIN information from another database on the same server, you can include the database name in the query. It's also possible to include data from an external data source in your query using a Linked server. See BOL for more information on this technique. One can also (sometimes) assume that the currently logged-on user is the owner or has rights to the table, so you can leave off the table owner's name as well. You can also leave off the table name if you are fetching data from only one table or the name is unique in all of the tables referenced in the FROM clause.

When folks migrate their code from Access/DAO/JET, they often find that the names assigned to their tables and columns don't seem to work. The problem is that if an object name has embedded spaces (as many Access examples include), the object needs to be framed in square brackets so that SQL Server's syntax validator understands how to parse the name. However, it makes a lot more sense to remove the embedded spaceseveryone will thank you later. For example, instead of naming an object "Production Parts", it would be better to use either "ProductionParts" (with CamelCase notation) or "Production_Parts" with an underscore character instead of the space.

Using Aliases for Table and Column Names

One technique I often use to shorten and simplify the T-SQL is to use an "alias" for the table name. This way I can use shorthand notation to refer to ambiguous column names. An example is shown in Figure 2.31. Actually, SQL Server does not require the AS clause, so you can alias a table name in the FROM clause simply by adding the alias after the table name. In this example, I alias the Publishers table as "P", the Authors table as "A", and the Titles table as "T".

Figure 2.31. Using the AS clause to alias table names.

In some cases, I want to rename the returned column, as when there are two columns with the same name in the SELECT. In this case, I can (should) return an alternate name for one or both columns. This is also useful when columns are generated by an expression, as shown in Figure 2.32. Note that if you don't provide an aliased name for the column (as I did for the AVG expression), SQL Server does not return any column name at all. This can confuse some of the tools.

Figure 2.32. Aliasing columns in a TSQL query.

Note that I forced the default database to "Biblio" by using the T-SQL USE operator. Note that USE changes the initial catalog/default database on the connection. This means that subsequent operations on this connection will use this new setting. This same rule applies whenever you use a SET statement (as I discuss later in this chapter). When the application (or connection pooling mechanism) closes or resets the connection, these settings are discarded and the connection is reverted to their initial settings.

Walking Through the SELECT Operation

In the example shown in Figure 2.33, SQL Server is requested to find the name, city, state, zip code, and telephone number of any publisher whose zip code is between "90000" and "99000".

Figure 2.33. Executing a simple SELECT statement.

[View full size image]

If any rows are found that meet the condition in the WHERE clause, SQL Server returns a resultset that contains:

  • Zero or 1 "rowsets". A rowset is simply a set of 0 to N rows of data that contain the requested data from the specified table columns.

  • One or more PRINT messages.

  • A message and error generated by a RAISERROR statement.

  • Zero or more OUTPUT parameters returned from a stored procedure.

  • An integer RETURN value from a stored procedure.

  • A "rows affected" value that indicates how many rows were returned by the query.

Understanding the WHERE Clause

Basically, the T-SQL WHERE clause specifies the condition(s) that must be met to permit qualifying rows to be included in the rowset returned from your query. It's very important and should rarely be left out of a TSQL statement. The WHERE clause can:

  • Be left off: If you leave off the WHERE clause, all rows in the table(s) are returned. Generally, this is not a good idea, so I usually take steps to limit the number of rows returnedexcept when I'm fetching data for a "look-up" table. For example, returning all valid state abbreviations (there are about 64 of these) makes sense, but fetching all 86,253 parts for a Boeing 727 does not.

  • Be simple: In this case, the rows are simply filtered and only a subset of the table rows is returned. For example, you could create a WHERE clause to return just those customers that owe you moneyor a lot of money.

  • Contain parameters: In this case, the WHERE clause logic can be altered each time the query is executed. This is perhaps the most common approach, as it permits you to write a single query (or stored procedure) that's used repeatedly. SQL Server is also optimized to expect parameter-driven queries, so it might be able to use a cached query plan. A common tactic is to include a "starting" and "ending" row in the WHERE clause. For example, you might code the following T-SQL query (as illustrated in Figure 2.34).

    Figure 2.34. Passing a parameter to a WHERE clause.

Using Parameters to Pass Literal Values

Eventually, you'll find that parameters play a pivotal role in T-SQL statements. Basically, they're placeholders for values supplied when the query is actually executed on SQL Server. When creating a parameter-driven SELECT query (or an UPDATE, DELETE, or INSERT statement), you replace specific values in the T-SQL with marked variables. No, you can't replace table, column, or other object names with parametersonly those values that can be expressed as literals can be parameterized. That's because the T-SQL compiler needs to resolve all object names at compile time.

In T-SQL, parameter variable names are always prefixed with the "@" symbol. The value of the parameter can be set in code, as in Figure 2.34, or passed to SQL Server, as when executing a T-SQL procedure stored on the serverthese are called stored procedures, and I'll talk about them later in this chapter.

The query parameter @StartingISBN, in Figure 2.34, is declared in T-SQL using the DECLARE statement (similar to a Visual Basic Dim statement). In this case, I use the SET statement to set its initial value, but it would generally be set by creating a Parameter object in ADO.NET and setting its value in your application instead of hard-coding the value. When ADO.NET is asked to execute the query, the Parameter Value is passed along to SQL Server to be inserted into the T-SQL as it executes. Sure, a query can have lots of parametersmost do. In Chapter 10, "Managing SqlCommand Objects," I'll discuss how to set up your ADO.NET code to handle all types of parameters, including input, output, input/output, and return value types.

Why is it bad to return lots of rows? Well, it's not reallynot all bad. If you plan to create a high-performance (or even reasonably well-performing application) or an application that scales, you'll want to limit the number of round-trips to the server, the amount of work the server has to do, and the number of rows that have to be sent to and from the server. These suggestions are only common sense. Sure, if you execute fewer queries, you'll have fewer round-tripsthat's good. However, if these queries take longer to execute or involve so many rows that the queries interfere with other application's resource demands and hog network resources moving the resultsets to your client, then it's bad. Consider that a sound design returns just the rows needed in the immediate to short-term future and fetches more only if needed. This approach means more round-trips but makes better use of the server and its resources.

Limiting the Number of Rows Returned

Before I lose focus, remember I'm talking about ways to limit the number of rows returned from the server. Let's look at a number of other ways to limit the number of rows returned besides using a WHERE clause expression:

  • Use a TOP[15] expression[16] in the SELECT (as shown in Figure 2.34). TOP can be coded in a number of ways, but basically, you specify the number of rows (SELECT TOP 10...) or the percentage of rows (SELECT TOP 20 PERCENT...) to return. Note that TOP returns the first n or percentage of rows based on the order in which the rows are ordered by the server. No, TOP won't work on your imported SQL Server database until you set the compatibility level to 9.0 or greater. I show you how to do this near the end of this chapter.

    [15] I've noticed that T-SQL auto-generated by Visual Studio and the SQL Server tools use TOP expressions to limit the number of rows returned.

    [16] SQL Server 2005 now supports any numeric expression (including a variable name) instead of only an integer. This means that you can pass a parameter to a TOP expression. TOP has also been expanded so that it can be used in INSERT, UPDATE, and DELETE statements.

  • Use the T-SQL SET ROWCOUNT to limit the number of rows returned by a query. For example, SET ROWCOUNT 20 limits the number of rows returned to 20. Use SET ROWCOUNT 0 to turn off this option (which returns all rows).

  • Use the new[17] TABLESAMPLE clause to limit the number of rows returned by a query. Unlike TOP, which returns only the first rows of a resultset, TABLESAMPLE returns rows selected from throughout the set of rows processed by the query. This means that the rows returned are a "sample" of the data returned from the query, so an ORDER BY would not be required to presort the data.

    [17] Introduced in SQL Server 2005.

IMHO

More rows are bad. Fewer rows are fast.

Using the WHERE Clause to Perform JOINs

When you need to return information from two or more tables, your T-SQL query must include WHERE-clause logic to identify the rows whose columns are to be included in the final rowsetthe product. This is called a "Join" operation. Typically, the tables being joined are "related". That is, they have primary key (the column[s] that uniquely identify a row) and foreign key relationships. For example, in the Biblio database, the Customers and Orders tables have a relationshipthere are 0 to N orders for each customer. This is called a "one-to-many" relationship. The Orders table refers back to the Customers table, with the CustID column as a foreign-key relationship.

Figure 2.35 is a database diagram created by SQL Server Management Studio (or Visual Studio's Server Explorer) that illustrates the primary key and foreign key relationships between the aforementioned tables. When Visual Studio generates T-SQL queries for you, it uses these defined relationships to conger the correct JOIN syntax. Generally, you won't have to write your own JOIN clause, as that'll be done for you. But if you do want to do so, the best way to learn how is to inspect the queries that Visual Studio (or SQL Server Management Studio) creates for you.

Figure 2.35. A database diagram illustrating the Customer, Order, Item, and Addresses relationships.

[View full size image]

In the early days, I also used the WHERE clause to indicate how tables were to be joined relationally. This Code and Date technique was easy to teach but did not always work with other relational engines. For example, one could code a T-SQL query to relationally "join" the Titles and Publishers tables by simply cross-referencing the primary keys (as shown in Figure 2.36).

Figure 2.36. Joining tables using a WHERE clause.

No, this syntax is no longer in vogue, but it's still supported and works (and is optimized) just the same (in most cases) as a JOIN clause. Sometimes, it's easier for developers to "see" the join using this syntax, so it crops up from time to time. Essentially, a JOIN instructs SQL Server to collect information from two or more tables and combines the result (the "product") into a single rowset that's returned to ADO.NET. Be carefulif the JOIN is incorrectly coded, you might end up with a "Cartesian product," which contains all rows from one table combined with all rows from another. If you end up with 400,000 rows from a couple of smallish tables, that's likely what has happened.

Coding TSQL JOIN Syntax

Coding a JOIN operation between two (hopefully) related tables defines how the SQL product (the rowset) is built by specifying which columns in the two tables being joined are "related." As I've said earlier, the two columns typically specify a foreign key from one table (the "left" table) and a primary key from another (the "right" table). Since SQL Server (and Visual Studio) adopted the SQL-92 syntax, we've encouraged code joins using the JOIN operator. Depending on how the JOIN logic is intended to work, you can code the JOIN in either the FROM clause (which specifies the tables to use in the query) or the WHERE clause (which limits the rows returned by the query). Joins are categorized as:

  • "Inner" joins: Typically, an inner join asks for specific rows from one table, where the primary key is "in" or "not in" the foreign key table. In other words, the columns specified in the join condition expression must match in both tables for rows to be included in the rowset product. For example, an inner join might fetch columns from the Publishers table where the PubID column in the Titles table matched the primary key in the Publishers table, as shown in Figure 2.37. This is how you fetch columns from two or more tables.

    Figure 2.37. A simple INNER JOIN returning data in related rows from Titles and Publishers.

    IMHO

    If your "simple" query takes all afternoon to run, you've probably created a Cartesian product. Don't despair; these aren't illegal outside of Texas.

  • "Outer" joins: These are a lot more complicatedthey're different, in that they're designed to return all of the rows from either or both tables based on the join condition. These are used in special circumstances, as when you need to locate rows in the left table that don't exist in the right table. For example, to find all Titles that don't have a known publisher city, you could code the T-SQL as shown in Figure 2.38. Note that, in this case, the JOIN syntax returns all rows from the Titles tablewhether or not there is a matching Publisher PubID.

    Figure 2.38. An outer join against the Titles and Publishers tables.

  • A Cartesian product is returned when each row from the "left" table is combined with each row from the "right" table. This is also called a "cross join", as it reflects your mood when you discover your rowset (which took 37 minutes to create and return to the client).

Coding a UNION Query

Suppose you have a situation where you need to return two or more rowsets from several tables in a single rowset. Sure, in most cases, a SELECT statement can join the tables, but when you simply want to return multiple rowsets as one, you can use the UNION statement. The UNION can append columns from expressions or literals as well. For example, when creating a pick-list query, it's useful to have the first row of the list show a "not selected" or NULL value without having to add code on the client to deal with it, as shown in Figure 2.39.

Figure 2.39. Using the UNION statement to append data to a rowset.

This technique is especially helpful when working with Reporting Services, as it does not accept more than one rowset. The only caveat here is that the datatypes for each column you append must match (to some extent) the datatype, and the number of columns must be the same as the columns in the original SELECT. Note that the columns in the resulting rowset are set by the first SELECT, so you might want to alias these (as shown in Figure 2.39).

Ordering the Rows Returned

When you use a SELECT statement to return rows from a data table, the rows are not returned in any particular order (that you can depend on). Even if a clustered index (which physically sorts the data rows as they are added) is defined for the table, SQL Server 2000 and later do not guarantee the order in which rows are returned. This is because SQL Server might perform "parallel" processing on your query and simultaneously extract qualifying rows from different parts of the table.

To define (and guarantee) the order in which rows are returned, you must add an ORDER BY clause to the SELECT statementor accept the rows in pseudo-random order (however, SQL Server decides to return them). The ORDER BY clause permits you to specify one or any number of columns (or combinations of columns) to sort by in either ascending (ASC) or descending (DESC) order. In the example shown in Figure 2.40, the ORDER BY clause sorts the data based on the publisher's name (PubName), and since no direction is indicated, ascending is assumed. It also sorts on the author's name (ascending) and the year published (descending).

Figure 2.40. Using the ORDER BY clause to order the returned rowset.

[View full size image]

Note that this query also returns the results of an expression (Year_Published-Year_Born) and labels the column as "YearsToPublish". Yes, this is easy to do, but when you try to get Visual Studio to generate UPDATE statements for this rowset, it gets confused.

When you install SQL Server, and again when you create a new database or column, you can define (using the COLLATE argument) how data is to be sorted by the ORDER BY statement. A collation defines the sort order to use for Unicode data (nchar, nvarchar, and ntext) and non-Unicode (char, varchar, text) data. The selected collation defines the order in which specific characters are stored and ordered when an ORDER BY expression is added to a SELECT or how expressions are evaluated.

Generally, each (human) language defines specific collating ordersfor example, French, German, Thai, and English all have different collating orders for some (or most) characters. When the data arrives at the client, the Windows locale selected on the computer is used to determine character ordering.

Dealing with Special-Case Query Problems

Most of the documentation shows you how to write simple queries, and Visual Studio will even help you write most of these. The problem comes when you're asked to do something out of the ordinary. Well, based on my experience, these "special cases" seem to take up quite a bit of my time to figure out. Perhaps, these scenarios will help shorten the time it takes to solve a few of these for you.

Managing Case-Sensitive Queries

From time to time, I've been asked to write queries to locate data (usually names) based on a case-sensitive expression. Don't confuse this query technique with "politically correct" or "person-sensitive" queriesthat's an entirely different kind of sensitivity. Case-sensitive queries differentiate between strings in UPPER- and lowercase. Suppose you want to find books whose titles contain the letters "DOS", but you don't want to include titles on "Pcdos" or titles about Mexican beer. Yes, this is a bit contrived, but in situations where you need a case-sensitive search, you might find it tough to find rows based on the character case.

Consider that when you install SQL Server, you have a choice of what collating sequence to use for all databases in the server. I discussed this option earlier in this chapter. Generally, when I install SQL Server, I don't configure it to be case-sensitive because if I do, each and every T-SQL query I and every other developer writes must match the required case of the T-SQL operators in addition to all database object names, including the database, table, column, view, rule, trigger, stored procedure, and every other object in the database. This means you would have to code:

SELECT MyCol FROM MyTable

instead of:

select mycol from myTable

In a non-case-sensitive installation, either query would compile and work the same way. I've had to troubleshoot sites that chose to install a case-sensitive server, and I can tell you it's a royal PIA to deal with (and that's not a primary interop assembly). Every example the customer tries seems to have syntax problems. Sure, most of the tools are careful about how T-SQL is generated, so they use the correct case and I try to adhere to these standards in my books, but it's tough to do sowe're only human. It's not surprising that many sites insist on non-case-sensitive installations and don't send flowers to DBAs that insist on it.

Fortunately, SQL Server 2005 and more recent versions permit you to specify case-sensitivity and other collation criteria as properties of the database, table, column, and querysettings that over-ride the server default. This way, you can code your T-SQL to specifically indicate how the server is to compare the values in any given expression, even though the server is set to not be case-sensitive. For example, you can specify several different attributes using a collation string.

  • Language: Each language recognized by SQL Server has its own sort order based on the alphabet used by that language. I won't bore you with the details, but suffice it to say that English collations are marked as "Latin" (for some reason).

  • Case Sensitivity: This setting determines whether to ignore case ("CI") or to observe case ("CS") when evaluating expressions.

  • Accent Sensitivity: This setting determines whether to ignore any accents ("AI") or observe accents ("AS") when evaluating expressions.

  • Kanatype: This setting determines whether to ignore any accents ("AI") or observe accents ("AS") when evaluating expressions.

Sometimes, you can help SQL Server resolve the query, and you won't need to write a query that specifies a special collating sequence. Let's walk through some approaches to managing case-sensitive queries.

  • In some cases, you might be able to use the UPPER or LOWER functions to force the text value in the left side of an expression to upper- or lowercase so that it matches the literal used in the right side of the expression. For example (as shown in Figure 2.41), you can force a character column to uppercase (as shown) or lowercase (using the LOWER function).

    Figure 2.41. Forcing the case of an expression using UPPER.

    [View full size image]

  • To instruct SQL Server to use a specific set of collating rules, you can include a COLLATE clause with a collate string in the query, as shown in Figure 2.42. Note how the first query does not return titles with "DOS" (or "Dos" in any case combination), as returned in the second query.

    Figure 2.42. Using a collating string to specify special case, accent, and other collation rules.

    [View full size image]

This approach can be used only for any char, varchar, text, and the equivalent Unicode types. That makes sense, as you shouldn't really care about the case of numeric values. The COLLATE clause is not needed if you apply collating rules to individual columns when the database object (like a table column) is first created or when you execute a query against the data.

Framing Dates and Literals

Whenever a T-SQL query includes a string literal, it has to be enclosed in single quotes, as shown in Figure 2.41, where I frame the literal 'HITCH%' in single quotes. I refer to these as "string framing" quotes. If you use an ADO.NET Parameter object to manage your query parameters, the framing quotes are handled for you automatically. One of the most common problems with framing quotes is called the "O'Malley" issue because it crops up when you have Irish or other surnames or other strings that include single quotes. There are several ways to deal with the O'Malley issue. One way is to remove all of the offending names from the database. This has been tried at various times in history with limited success. Another (more acceptable) way is to let ADO.NET handle it for you by using Parameter objects to manage strings passed into SQL queries.

If you must code your T-SQL hard-coded queries, you'll have to make sure that any strings that include (or could include) single quotes are handled a bit differently. To pass a single quote inside a string destined to be sent to SQL Server, you need to double the single quote. For example, "O'Malley" becomes "O''Malley". Yes, that's two single quotes, not a single double quote. Confused? Well, remember that it's best to let ADO.NET handle these issues for you.

Note that dates also require special handling. SQL Server also frames dates with "#" characters or single quotes. Dates can be passed in a variety of formats, as SQL Server is smart enough to parse them (more or less) correctlyas long as you take them out of the husk and express them in U.S. format (mm/dd/yyyy). Yes, in the U.S., we use month, day, year notation (which is admittedly pretty strange to data processing types who like data to be stored in a more orderly fashion). In most of the rest of the world, other sequences are used, but SQL Server always stores dates in U.S. format. When you test for a date (as shown in Figure 2.43), if you don't strip off the "time" part of the date, it's tough to get an equality expression (X = Y) to work. That's why I either strip off the critical parts or use a BETWEEN expression to search for an acceptable range of dates.

Figure 2.43. Framing a date passed as a string literal.

[View full size image]

As with single quotes, if you need to include special characters (like double quotes, arithmetic operators, and other special characters) in your string literals, you'll need to double them up to get the T-SQL syntax parser to accept them.

Before you try to perform date arithmetic, be sure to check out the "date" functions in T-SQL. There are functions to add, compare, parse, and assemble dates. See DATEADD, DATEDIFF, DATENAME, DATEPART, and the other date functions for more information. Sadly, there are no date functions available to help some of you single folks find someone to go out with on Friday night.

Concatenating T-SQL Expressions

One technique you're going to see in a number of online examples is concatenated T-SQL. For example, when you want to create a complex query but haven't learned how to use the Command object's Parameters collection to manage query parameters, you'll likely try to simply concatenate the arguments together. This is not that terrible, as long as the values are all machine-generated, as shown in Figure 2.44.

Figure 2.44. Creating a concatenated TSQL query.

First, you should learn to avoid use of the concatenation "&" operator in Visual Basic (the "+" in C#). It's extremely inefficient. Peter hooked me on the String.Format function some time ago, and I've pretty much converted my concatenation code to use it almost exclusively. With this conversion, the statement now looks like Figure 2.45.

Figure 2.45. Using String.Format to concatenate TSQL strings.

Okay, this is certainly not a big change. However, what if you want to pass in the value from a TextBox or other UI control? What impact does this have? Well, you might be tempted to simply substitute the TextBox.Value for the hard-coded value (1947)something like Figure 2.46.

Figure 2.46. Concatenating from a user control.

Notice that I even cast the Text value to an integer, even though the String.Format would have handled that conversion for me. So what's wrong with this approach? It seems clean enough. The problem is that it opens your application to a "SQL injection" attack. No, I'm not going to show you how to hack into a site that uses this approach. Suffice it to say that it's really easy to do and it's documented all over the Internet. Let's take this exercise one step further and add T-SQL to the WHERE clause to filter for a specific Author by name. Take a look at Figure 2.47.

Figure 2.47. Using a string parameter in a WHERE clausewithout a parameter.

In this case, we add another argument to the String.Format expression and fill it with the value from a string captured from another TextBox. In this case, your application will workright up to the point that someone tries to find Mr. George O'Malley. Remember that issue I discussed when talking about framing? Notice that the String.Format expression includes single framing quotes, but that does not correct the problem of a string being concatenated that includes a single quote.

When I get to Chapter 10, I'll revisit how to build a Command object with a Parameters collection that can handle both issueswithout additional code.

Coding WHERE-Clause LIKE Expressions

Many application scenarios need to return rows based on an expression that helps locate rows in the table based on a "soft" string comparison. Since SQL Server was "born" in and around Berkley, California, you can imagine how the developers were accustomed to hearing and using the word "like" while describing like something, or like anything. Anyone with teenage sons or daughters in the last two decades can attest to the popularity of this manner of speechespecially on the West Coast of the U.S. In a single half-hour car trip, a teen could, like, use the word "like", like, a thousand timeslike, okay?

I used a LIKE expression in Figure 2.42 to locate titles that contained the characters "dos". Suppose you need to locate all rows in a table where the city name starts with the string "San". This would return "San Francisco", and half the cities and hamlets in the Southwest U.S. would qualify. You can use a LIKE expression (WHERE City LIKE 'San%') to implement a query whose WHERE clause filters in this manner. Note that in T-SQL, the "%" is the wildcard operator[18]. If you place the wildcard character at the end of the LIKE expression, SQL Server can usually leverage an available index to locate the rows (in this case, I would need an index on the City column). However, if you code your LIKE expression to find the string anywhere in the column (WHERE City LIKE '%San%'), the server usually has to search the entire tableit can't effectively use the table's index.

[18] In Microsoft Access/JET and DOS, the wildcard character is "*".

There are a bevy of options available to you when you use the LIKE expression, as shown in Table 2.2.

Table 2.2. Using the Like Expression to Qualify Rows

Wildcard Character

Description

Example

%

Matches string of zero or more characters.

WHERE City LIKE 'San%' or Part LIKE '%Upper%'.

_ (underscore)

Matches single character.

WHERE Weapon LIKE 'Cat_pult', which would return "Catrpult" or "Catapult" but not "Cataupult".

[ ]

Matches single character within specified range. ([^a-f]) or set ([^abcdef]). The "^" is a "Not" operator.

WHERE Author LIKE 'de^l%' (returns authors with names beginning with "de" and followed by any character except "l". This would return "Demure" but not "Delasandro".

Sure, you can pass the LIKE expression to a query as a parameter. As I'll discuss in Chapter 10, you can build an ADO.NET Parameter object to accept the LIKE expression as a string (including the wildcard characters). ADO.NET handles the framing quotes for you.

Coding IN Expressions

Another powerful way to focus your query on the desired rowset is to use the IN expressions. This operator permits you to specify a fixed set of values to test against at runtime. Yes, this set can point to a table, a function, or a literal string. The syntax is fairly simple (as shown in Figure 2.48). In the example, the first query uses a fixed set of values. No, these values cannot be provided with a parameterthey must be available when the query is compiled. In the second example, I use a SELECT statement to return the list of acceptable values. Yes, the list of IN clause items can be a stringin this case, I provided a couple of date strings.

Figure 2.48. Examples of IN expressions.

[View full size image]

No, the IN, LIKE, and BETWEEN operators don't have to be capitalized, as I (try to) showunless you've configured a case-sensitive database (please don't).

One other point about IN expressions that needs to be mentioned before I move on: No, you can't pass a parameter to the IN expression. That is, if you wanted to let your application choose the available IN expression values (as the set of dates passed to the IN expression in Figure 2.48), you could not pass them as a query parameterthey must be hard-coded when the routine is compiled by SQL Server. I discuss a way to get around this issue in Chapter 14, "Creating and Managing Reports," by creating a T-SQL (and CLR) function to convert a parameter containing a delimited string to a table that can be consumed by the IN expression.

Coding TSQL BETWEEN Expressions

When you need to test for a range of values, you can use the BETWEEN operator, as I used in Figure 2.43. The BETWEEN operator is handy when you're testing for a date value (as in the example), as it can "bracket" the values more easily. The syntax is easy and can also be "notted"that is, you can code "<test expression> NOT BETWEEN <beginning expression> AND <ending expression>". BETWEEN returns a Boolean TRUE if the value of the test expression is greater than or equal to the beginning expression and less than or equal to the ending expression. Sure, you can use the ">" and "<" instead of BETWEEN to create a more focused expression.

I use BETWEEN quite a bit when testing for date values because the date datatypes (datetime, Smalldatetime) include the time, which can confuse a simple comparison test. For example, if I want to test for equality on a specific date, it won't match unless the time is also exactly the same. Of course, you could (perhaps should) go to the trouble of using the T-SQL DatePart functions to break off the specific part of the date for which you're testing. See BOL for more information on DatePart and the other date-handling functions.

Handling NULLs

When a column contains a value, it must comply with the datatype specified for the column. However, if the column cannot contain a value because it's simply not known, you might be able to assign NULL to the columnassuming you've defined the column to permit NULLs in the table definition. For example, when you're filling out a form and it asks for "date married", not all of us can provide a dateespecially those trying to remain single long enough to graduate from high school. There are many situations where assigning a column to NULL makes senseand some that don't. When you define a table's PRIMARY KEY or IDENTITY property, it cannot be permitted to have a NULL value. Any other critical information in the table should not be permitted to be set to NULL. When your table has a primary key/foreign key (a parent/child) relationship, you cannot permit the parent value to be set to NULL. That is, there must be a valid parent (primary key value) for each child row.

Handling and Comparing NULL Values

There are several special rules when it comes to handling NULLs. First, remember that a NULL is not an empty string (""), a zero, or blankit's a definitive "state". In addition, you can't simply compare a value against a NULL. "IF 5 = NULL" does not make sense, just as comparing a column value against NULL. If you need to determine whether a column contains a NULL value, you must use the IF NULL or IS NOT NULL T-SQL syntax. When you want to assign a NULL value to a column in T-SQL, you can use the keyword "NULL" in the INSERT or UPDATE statementor by leaving the column value out of the INSERT statement. In this case, the INSERT operation first checks to see if there is a default value defined for the columnin which case, it's used; otherwise, the column is set to NULL. When searching a table by referencing a column that might contain NULL values, SQL Server ignores any NULL columns and does not include them in the rowset membership. When I get to Chapter 10, I'll discuss how to handle NULLs in ADO.NET queries and parameters.

How SQL Server handles NULL comparisons is determined by the connection-specific SET ANSI_NULLS setting:

  • When ANSI_NULLS is ON and one or more of the expressions in a comparison is NULL, the result is always NULL. That's because any comparison of a value with NULL is meaningless. It's like asking "What was the date that you died?" The side effect of this setting is that if you compare a column that contains a NULL (IF DateMarried = NULL), it always returns NULLregardless of the contents of DateMarried. In this case (which is the default setting), you have to test for NULL values using the IS NULL or IS NOT NULL clause.

  • When ANSI_NULLS is OFF, you can compare columns that might contain NULL values with NULL and return a True or False. That is, (IF DateMarried = NULL) would return True if DateMarried contains a NULL.

You can change the state of the ANSI_NULLS switch using the SET statement in your T-SQL or use the Query Options properties dialog (shown in Figure 2.49) to change it.

Figure 2.49. Setting ANSI options using the Query Options dialog.

[View full size image]

It makes sense that computations involving NULL values also return NULL. Consider that if TotalSales is NULL, the expression (TotalSales * 1.25) should also return NULL. It's like asking the head of your sales division, "Hey, Sally. I hear that the sales figures aren't in for the fourth quarter, but if they were up 2.5%, what would that be more than last year?" If she answers, you might need to find another sales managerpreferably one that had not worked for the government.

Okay, so how does one code a T-SQL expression when ANSI_NULLS is ONas it will be unless you over-ride the default? It's not that hard. Take a look at the example in Figure 2.50. In this case, I'm testing the Orders and Items tables to see if any of the orders were not shipped even though there were confirmed orders. Note the syntax for the "IS NOT NULL" case. This T-SQL code was generated by Visual Studioit negates the IS NULL expression instead of using the IS NOT NULL syntax. You're free to use either technique.

Figure 2.50. Testing for NULL values in a TSQL SELECT.

Using the sql_variant Datatype

Any Visual Basic developer who has transitioned to Visual Basic.NET can tell you how the Variant datatype has changed how data access problems can be addressed. In situations where the data being passed from tier to tier or even in-process is unstructured or "fuzzy," the Variant datatype can be a good choice. This is the datatype most favored by political partieseven George W. Bush made reference to "fuzzy" data in the Presidential debates; making an obvious reference to Variant data. In Visual Basic, the Variant datatype permits developers to specify a program variable that refers to virtually any type of data, be it a number, string, picture, or object. Visual Basic stores the data and additional data description information to help Visual Basic and the programmer manage the information. I won't get into how efficiently (or inefficiently) Visual Basic handles Variants, but consider that the new Visual Basic.NET framework no longer supports the Variant datatype. (It's replaced with the Object datatype.)

Now, suppose you have a situation where your application calls for additional information to be stored for a given customer, but you don't really know ahead of time what that information looks like. However, you do know it's going to either be a number (an integer, float, or decimal), a string, or perhaps even a Boolean value. (It might also be a binary large object [BLOB], as would be stored in a TEXT or IMAGE datatype, but SQL Server sql_variant datatypes don't let you store BLOBs, so let's pretend that these are not an option for your design.) One approach you might take is to create a table that looks something like Table 2.3.

Table 2.3. Attempting to Support Multiple Properties for Objects

ObjectID

Prop1

Prop2

Prop3...

...Prop100

101

123

Null

Null

Null

101

Null

'George'

Null

No

102

Null

'Fred'

Null

Null

103

Null

Null

22.50

Null

This table defines 100 columns, each specifying a "property," each with a different purpose and each with a use-appropriate datatype. The ObjectID column provides a relational reference to a unique entitypossibly an object in the database or perhaps just a customer. If you wanted to determine what the Prop3 value was for a particular customer, you could easily fetch it using a simple SELECT. However, the sheer width of this table would make a relational database designer cringe or perhaps bring C. J. Date[19] back from the grave. As you can imagine, this table is mostly filled with nothing (Null values), so it's not particularly efficient to manage. And what if you have to define more (and more) properties as your business rules evolve? What if there is more than one row that defines a specific property or if the combination of several properties exhausted the row capacity of SQL Server (a little under 8K)? From here on, you're on your own.

[19] The "father" of relational databases.

Another Approach

SQL Server 2000 and later (thankfully) support Variantsat least, in a way. While the implementation is not the same as that supported by the Visual Basic 6.0 Variant datatype, the sql_variant SQL Server datatype is flexible enough to solve our rule-morphing problem. Since sql_variant can take on (almost) any SQL Server datatype, our customer data property table (as shown in Table 2.4) now looks a little different.

Table 2.4. A "Sparse" Table to Manage Customer Data Properties

ObjectID

Property

Info (sql_variant)

101

'Prop1'

123

101

'Prop2'

'George'

102

'Prop2'

'Fred'

103

'Prop3'

22.50

In this case, I define the table with three columns: one to identify the object, another to name the property being managed, and a third to hold the data for that property. This is an example of a "sparse" table. This permits us to define as many properties as the current business rules dictate. Since the sql_variant "Info" column can hold any (non-BLOB) datatype, we can easily store whatever information that presents itselfeven if we didn't anticipate that type of data during the design phase. The challenge here is to write a front-end application that knows how to adapt itself to this ever-changing set of properties.


Creating and Querying Views

SQL Views have been around for some time, and if you've worked with any serious SQL engine, you've doubtless used Views to dereference complex queries. Of course, SQL Server implements Views according to the ANSI standards, but it also takes a number of big steps beyond those modest standards. Let's step through the "view" features you're most likely to use.

T-SQL Views

A T-SQL view is basically a query containing a SELECT that returns a single rowset from one or more tables or other viewsit can be considered to be a "virtual" table or (simple) stored query. No, when a View is referenced, the rows are not persisted any differently than when they're fetched with a SELECT. Once created, a View can be used in T-SQL syntax just as you would use a table.

Why would one want to use a T-SQL View instead of just coding a SELECT? Well:

  • A view can make writing your code (both T-SQL and ADO.NET) easierjust like using a macro or subroutine call to hide underlying complexity.

  • Since a DBA can (and should) hide underlying tables, a user can be granted access to a View while not having access to the base table(s) it dereferences.

  • Expression values and aggregates are resolved as the data is fetched instead of being stored in a table.

  • A view can be updateable (but only one of the base tables at a time).

  • A view can be indexed to improve query performance.

  • Unlike a base table, a View consumes no additional storage.

Creating Views

Visual Studio and SQL Server Management Studio can both create Views using the Server Explorer and the Object Explorer (see Chapter 5 for details)however, Views are not "compiled" or stored anywhere except in the system catalogs (as T-SQL). When referenced, the query optimizer simply uses the View definition as a compiler would insert macro code, so they have no impact on performanceother than developer performance.

IMHO

A View is referenced in your T-SQL code just like any other table. In this case, it can make sense to use SELECT *.

Let's step through some examples. I'm going to show you the T-SQL code used to create a View, as the Visual Studio IDE tool to do the job is very intuitive. In the example shown in Figure 2.51, I start by testing to see if the View is already defined. This is accomplished by executing a SELECT against the system object catalog (sysobjects). If the View exists, I execute a DROP VIEW to remove it from the catalogs. Sure, you can use an ALTER view to change an existing VIEW, but this script syntax is typical of what you see when DBAs set up scripts to rebuild the database from scratch.

Figure 2.51. Creating a TSQL View.

Next, I create the T-SQL View. This view returns selected columns from the Authors table, including an expression-generated column. The expression computes the author's age and returns the value as a column named "YearsOld". The View's WHERE clause restricts the rows returned by filtering out authors without a stated Year_Born or whose age is beyond reason. This filters out irrelevant and test data.

The View also resets the default collating sequence so the rowset returned is set to a case-sensitive and non-accent-sensitive order. This means expressions executed against the columns returned will be case-sensitive.

Note that a T-SQL View cannot include an ORDER BY clause, but it can include special collation instructions, as shown in Figure 2.51. There is nothing to stop you from sorting the data in the SELECT statement that returns the rows, as shown in Figure 2.52.

Figure 2.52. Accessing a View with a SELECT statement.

If the View returns a lot of rows you might get better performance by applying an Index on the View. This was new for SQL Server 2000, so you might already be aware of this feature. However, be sure to consult BOL[20] before attempting to set up an indexed viewthere are a number of limitations that you should consider before wasting a lot of time. For example, an indexed View definition cannot contain:

[20] See "Designing Indexed Views" in Books Online.

  • TOP and most aggregates expressions, DISTINCT

  • BLOB columns

  • UNION

  • ORDER BY

  • An expression that generates a nondeterministic value

  • And a lot more...

The example shown in Figure 2.53 shows the T-SQL code needed to create an index on a View. Notice that the View itself has been simplified. I had to remove the age expression. SQL Server forced me to create a UNIQUE CLUSTERED index on the Au_ID column, as it's the only unique column in the table. As the BOL documentation suggests, Indexed views won't help in all situationsas a matter of fact, it looks like their usefulness is fairly limited in scope.

Figure 2.53. Creating a View and an index on the View.

Database Snapshots

Peter and I are very excited (if that's the right word) about Database Snapshots. This new feature permits a developer (or DBA) to "snapshot" the database in time so that one can get a stable view of the database without the overhead of creating another parallel database. As the main database is changed, the Snapshot version keeps its own copy of the original pages as they are modified. This can be really handy if you need to recover from an accidental change simply by reapplying the original pages (stored in the Snapshot database) to the main database.

We also think that this approach is interesting for reporting applications that need a "static" copy of the databaseagain, without the overhead of an entirely separate database against which you run your reports.


Implementing Business Rules

As discussed in Chapter 1, business rules are often an importantnay, criticalcog in your application. Business rules, by their very nature tend to morph over time to reflect changes in the business climate, marketing, sales, and relationships with customers. If these are hard-coded in your applications, they must be redeployed when they change. At EDS, much of my data in the Part B healthcare support team was spent changing applications to reflect the ever-changing federal regulations that governed how we adjudicated claims. These business rules could be in the form of default, minimum, maximum, usual, prevailing, or arbitrary values, as well as masks (for specific numeric layouts like phone numbers and social security numbers) and other inter-column data relationships. For example, a claimant could have his left leg removed below the kneebut only once. They're permitted only one appendicitis operation, one hysterectomy, and only two radical mastectomies.

Sure, your business rules probably won't have to be quite as grizzly, but they're nonetheless important. As you deal with hotel rooms, airline seats, customer discounts, and sales commissions, there are bound to be rules your business establishes to manage these values. These rules check to ensure that the rules meet the established criteria for validity and to maintain referential integrity in the database. That is, suppose you have a list of valid parts suppliers. When you add a new order, the Supplier column should contain a valid SupplierID drawn from the ValidSuppliers table. Referential integrity makes sure that's so. Data validity business rules ensure that QuantityOrdered is within the reasonable range of 1 to 50 when the part ordered is a 50-lb. anvil.

Just how (and in which tier) you implement your business rules is up to you and your application architect. Some feel that applications are not a good place to validate these rulesit can mean that the application must be recompiled, tested, and redeployed each time the rules change. That's simply unacceptableeven when the applications are redeployed through one-touch or via a web application. Some think that business rules should be implemented in a middle-tier object layer. Sure, this is a good idea, in some casesespecially if your application starts getting large. However, in this case, you'll likely spend quite a bit of resources designing and building objects, training everyone on your team (now and for the foreseeable future) how to use them, as well as fighting unique "object" issues that simpler designs don't encounter. Some think that business rules should be implemented in stored procedures where they become a part of the database and can be managed centrally. This assumes your team knows how to write and manage shared stored procedureswhich is not a universal truth.

I'm of the opinion that business rules should be executed as close to the point of data collection as possible but designed so they can morph automatically when the rules change. This means creating an infrastructure within the application to capture the latest rules and modify the client-side variables that they govern. This approach can be implemented any number of waysby reading the business rules from SQL Server database tables, by reading them from shared files, or by loading them from a SQL Server Everywhere databasemy favorite solution.

Implementing Business Rules with Extended Properties

Another approach that I've advocated for years uses a unique SQL Server feature called Extended Properties. BOL clearly describes how Extended Properties can be created, modified, or deleted, but there is little guidance beyond a few suggestions on how these properties can be used. That's the point. Their use is limited only by your imagination. I think Extended Properties could easily be used to help implement automatically morphing business rules.

Extended Properties can be assigned to virtually any SQL Server object, including databases, tables, columns, views, rules, triggers, stored procedures parameters, constraints, and indexes. You can create Extended Properties using system-stored procedures or through SQL Server Management Studio (as shown in Figure 2.54)which is far easier to use.

Figure 2.54. Viewing or managing Extended Properties.

Extended properties are added, updated, and deleted using the dialog shown in Figure 2.54 or by calling three system stored procedures:

  • sp_addextendedproperty: Adds a new extended property to a database object

  • sp_updateextendedproperty: Updates the value of an existing extended property

  • sp_dropextendedproperty: Drops an existing extended property

Once the Extended Properties are stored in the database, your application can retrieve the values by name using a system function, FN_LISTEXTENDEDPROPERTY.

At first blush, these stored procedures can be somewhat daunting, but once you understand how the parameters address individual objects or groups of objects, they're pretty simple to use. Books Online (BOL) provides a table (shown in Table 2.5) that describes what values the stored procedure parameters can be set to and what combinations are valid:

Table 2.5. Valid Extended Property Stored Procedure Parameters

Level 0

Level 1

Level 2

User

Table

Column, index, constraint, trigger

View

Column, INSTEAD OF trigger

Schema-bound view

Column, index, INSTEAD OF trigger

Stored procedure

Parameter

Rule

<None>

Default

<None>

Function

Column, parameter, constraint,

Schema-bound function

Column, parameter, constraint

User-defined data type

<None>

<None>

Take a look at the sp_addextendedproperty stored procedure. It accepts eight parameters. Figure 2.55 illustrates a typical example that sets an Extended Property named "caption" with a value of "Employee ID" on table (Employee), column (id), and owned by user "dbo".

Figure 2.55. Adding an extended property to the Employee.ID column.

The first two sp_addextendedproperty parameters name the Extended Property and the value. Since the Value argument is a sql_variant, you can pass anything except a BLOB (see the earlier discussion and BOL for more details on the sql_variant).

The six remaining sp_addextendedproperty parameters point to the objectin this case, a specific column. Notice how the fifth sp_addextendedproperty parameter names the "Level 1" object type ('Table', 'View', 'Rule', etc.) and how the sixth "Level 2" parameter names the object itself ('Address' in the previous example). For example, to add an Extended Property to a SQL Server table object, you need to pass two levels to sp_addextendedpropertythe user and tableas parameters. The remaining level 3 parameters can be omitted. For example, to set a couple of Extended Properties on the Publishers table, you could code a batch as shown in Figure 2.56.

Figure 2.56. Adding two Extended Properties to the Publishers table.

[View full size image]

An Easier Way

While you can use system stored procedures to manage Extended Properties, it's far easier to use SQL Server Management Studio. To view or manage Extended Properties, right-click the object in the object tree view and click "Properties". Next, select "Extended Properties", as shown in Figure 2.57. In this case, I'm addressing the Publishers table in the sample Biblio database.

Figure 2.57. Managing Extended Properties using SQL Server Management Studio.

[View full size image]

The Extended Properties dialog is intuitive and easy to use. However, consider that the dialog does not care if you add more than one property with the same name. Since the entries you make are not committed to the database until you press "Apply" or "OK", duplicate entries are not picked up until the dialog attempts to update the database. Notice that the Value field accepts Unicode strings. This same dialog is used to add Extended Properties to any of the supported objects.

In your application, you'll want to retrieve the Extended Properties and use the values to morph your client-side business rules or simply change the captions for the data columns being managed. SQL Server provides a system-level Function for this purpose: FN_LISTEXTENDEDPROPERTY. Its parameters follow the same hierarchical pattern as the stored procedures used to manage Extended Properties. That is, to return a specifically named Extended Property, you pass all three levels to the function. By passing "NULL" or "Default" instead of one or more levels, the FN_LISTEXTENDEDPROPERTY returns all of the Extended Properties under the specified level. For example, the example shown in Figure 2.58 returns all defined Extended Properties for the Publishers table (in the Biblio database).

Figure 2.58. Returning all of the Extended Properties for the Publishers table.

[View full size image]

One of the problems I expect you'll have with SQL Server Management Studio's query editor (I know I've hit it often enough) is that if you don't specify the correct default database before executing, your query won't do what you expect. It's good practice to add the USE statement at the head of your queries so they always do what they're toldto the correct database.

The first query in the example in Figure 2.59 returns the database object's Extended Properties but none of the database's subobjects (such as Tables). The second and third examples return table and column Extended Properties for the Publishers table.

Figure 2.59. Using fn_ListExtendedProperty to return selected Extended Properties.

[View full size image]

Other Extended Properties Issues

Another, more complex challenge faces developers who use Extended Properties: discipline. That is, there is no standard way to define these name-value pairs. The Extended Property names aren't case-sensitive (unless the database is set up to be case-sensitive), but if your company does not standardize on a naming convention and rules that dictate how and where Extended Properties are defined and who can define them, you'll be welcomed into your own world of "DLL hell." This is the same challenge facing developers building COM components or stored proceduresany shared code technology. Other developers who use and reference these objects hard-code the names given to these COM properties, methods, SQL Server stored procedures, and their arguments. These shared-code reference names cannot be permitted change once the class, object, or stored procedure is published and developers begin to code against them. While it's easy to make these changes (the latest GUI tools make it easier than ever), your team's discipline and training must make sure that these names aren't rearranged, their data-types aren't changed, and the arguments remain the same. While it's possible to add arguments to existing objects, you must maintain backward-compatibility with the previous version. But you knew that.

Extended Properties can easily make a significant impact on your application's ability to better manage business rules, how your applications (or web pages) deal with data-driven displays and validation rules, and much, much morelimited only by your imagination. I envision use of these properties to manage business rules or user-interface parameters for the data. Imagine being able to set minimum, maximum, default, today's value, suggested value, field masks (###,###.00), localization strings or any number of attributes that can change from time to time. When the application first starts, it can read the Extended Properties and morph itself to the new properties.

Implementing Business Rules with T-SQL Rules, Constraints, and Defaults

Another, more traditional approach to handling business rules has been around since the earliest days of SQL ServerT-SQL Rules. While (again) this approach is not in vogue, it's still supported (at least, until Microsoft tries to pull it) and a very understandable way to handle this problem. As I see it, as Microsoft has attempted to conform to SQL ANSI standards, many of the traditional (and well-understood) techniques have fallen from favor. Yes, in years to come, Microsoft might find a way to replace Rules, but I don't see that happening anytime soon. I would keep using them as long as they make sense. Microsoft says that Rules are provided in SQL Server 2005 for "backward-compatibility." Microsoft wants developers to use CHECK constraints as the "preferred" and "standard" way to restrict column values. They're rightonly one Rule can be applied to a column, while any number of CHECK constraints can be applied. But CHECK constraints can't be applied to User Defined Typesas you can with Rules. Rules and CHECK constraints are also managed differently, as the following discussion highlights. Yep, I'm still a Rules fan.

T-SQL Rule operators are not that well documented in BOL (although they used to be). Let's step through the process of creating T-SQL Rules and how they can report failures back to your application. Think of a Rule as a WHERE clause for a new row's column(s). That is, if the conditions set by all column Rules are met (as in a WHERE clause), the row is accepted into the table. Rules can be associated with a specific column or a T-SQL User-Defined datatype (UDT) assigned to a column. No, these UDTs are not the ones you can create with the Common Language Runtime (CLR) that I discuss in Chapter 13, "Managing SQL Server CLR Executables" these are the old-fashioned user-defined datatypes that are simply aliased SQL datatypes. I discuss T-SQL UDTs in the next section, so you might want to skip down there to get a feel for how they work before I go onjust remember to return here when you're done.

When you assign a rule to a T-SQL UDT, you define the special criteria that must be met whenever a column is defined with the UDT. For example, you can create a UDT for the Discount column and assign a Rule to the UDT (instead of directly to the Table). This rule might limit the values applied to the Discount column to values between 0 and 99. As you can see, this goes well beyond simple type checking. Sure, it's important to test to ensure that the value supplied is an integer, but that still permits invalid values from being passed to the Discount column. Yes, a Constraint can also be applied to the column to perform the same check. However, when the business rules change, you must revisit each and every column Constraint that validates the Discount value. I've mentioned this shortcoming to the SQL Server team on several occasionsthey say their hands are tied by ANSI requirements.

Creating Rules

Ah, Microsoft has disabled the ability to create new Rules from the SQL Server Management Studio IDE and they're not even listed as visible with Visual Studio. This means your only option if you want to create a rule is do it the hard waythe way we created Rules for yearsusing a T-SQL editor. Since rules are very easy to write, this should not be a problem. Do you think they're trying to hint at something? I expect so.

When you want to create a rule, simply execute the CREATE RULE statements, as shown in Figure 2.60.

Figure 2.60. Creating a TSQL Rule.

Once the Rule is created, it can be bound to any specific column in the databasebut only one rule can be applied to a column. This means if the logic in the Rule needs to use more complex validation logic, it must be included with AND or OR expressions. No, a Rule cannot reference any other column in its logic. That might be nice, but that's really the prevue of a Triggerwhich I discuss later in this chapter. Note that the variable (@Discount), in the example in Figure 2.61, can be anythingit just has to start with the "@" character.

Figure 2.61. Testing for a valid ZIP code layout.

[View full size image]

Rules are handy for making sure a column conforms to a specific pattern as well. For example if you want to ensure that a U.S. postal code (a ZIP code) conforms to postal regulations, you can create a Rule, as shown in Figure 2.61.

Binding Rules to a Column

Okay, now that you have a defined set of Rules, you need to decide how to bind them to the columns to be validated. This can be a lot of workespecially if the business rules implemented in the T-SQL Rule changes from time to time. To address this problem, most DBAs script their Rules and Rule Binding operations, and when it comes time to change the business rules, they alter the script and change them all en masse. No, you can't bind a rule to BLOB, CLR UDTs, or TimeStamp columns.

To bind a Rule to a specific column, you must execute the sp_bindrule system stored procedure, as shown in Figure 2.62.

Figure 2.62. Creating and binding the ValidDiscount rule in a script.

Okay, so what's a script? Well, hold on for a few pagesI'll show you how they work a little later. Suffice it to say they're simply a set of T-SQL commands in a single file that are executed in sequence. Each statement is separated from the previous by the "GO" operator. In this example, we make sure the default database is "Biblio", create the Rule, and bind it to the Customers table Discount column. Before I go on, let's talk about User-Defined typesthe ones that are managed by TSQL.


User-Defined (Aliased) Types

User-defined datatypes (UDTs) have been around since the earliest days of T-SQL. They're used to permit T-SQL developers to define a custom datatype that has specific Rules assigned there. Beyond that, they really serve no other (useful) function. You'll also find that the SQL Server BOL does not have much on aliased UDTsit's all been cobbled together with CLR UDT.

A T-SQL (aliased) UDT is just thatan object that dereferences one of the system T-SQL system datatypes. The SQL Server Management Studio IDE includes the ability to create T-SQL aliased UDTs in the IDE, but it's really pretty easy to do in T-SQL, as shown in Figure 2.63.

Figure 2.63. Creating a User Defined (aliased) T-SQL Datatype and Binding it to a Rule.

Looking a bit more closely at this script, we can see that the type "DiscountUDT" is created as an alias to the T-SQL float datatype. When this UDT is used in a table definition, any attempt to change the data in the column has to pass the criteria defined in the Rule bound to the UDT. Make sense? Notice that the @futureonly argument is set to 'futureonly'. This means that existing data in the database does not have to conform to the new Rule. Sure, if you want to revalidate existing data, set @futureonly to NULL. In this case, any column bound to the Rule or any column defined with the DiscountUDT and bound to the Rule is rechecked. As a result, the sp_bindrule can fail to execute until the database is fixed to make it compliant with the new Rule criteria.

That's Rules in a nutshell. Of course, Microsoft wants you to use CHECK constraints, and if you're worried about going out on a limb that Microsoft might saw off, you'll need to use CHECK constraints instead. No, they aren't as flexible as Rules and UDTs with Rules, but they're standard and, theoretically, are going to be supported longer. Hopefully, you'll be retired by the time Rules (or CHECK constraints) are dropped for something newer and better.


Implementing Constraints

SQL Server implements a whole family of constraints. These are used for a variety of purposes but generally are there to make sure your database integrity is maintainedboth data integrity and referential integrity. Yes, constraints can be used to enforce business rules, and I alluded to that earlier in this section. SQL Server supports the following types of constraints:

  • NULL (or NOT NULL): This constraint specifies whether a column can accept a NULL value. NULL means that the data is not known. I discuss NULLs later in this section.

  • CHECK: As discussed earlier, this validates data integrity by limiting what values can be applied to a column. These are defined when a table is first created or added later with the ALTER TABLE function.

  • UNIQUE: These constraints ensure that a specific column is unique within the scope of a table. This prevents SQL Server from adding a new row that has the same (or NULL) column value.

  • PRIMARY KEY: This constraint defines the column(s) that make a row uniquebut in this case, the primary key cannot be NULL. This means no two rows in a table with a PRIMARY KEY constraint assigned to one or more columns can be the same.

  • FOREIGN KEY: This constraint indicates that a value in the designated columns must exist as a value (usually a primary key) in another table. This constraint is used to help enforce referential integrity between tables related with primary key (PK)/foreign key (FK) relationships. For example, when the Customers and Orders tables have a FOREIGN KEY constraint defined, you can't create a new row in the Orders table without referring to an existing Customers table column (usually the PK like CustID). I discussed these relationships earlier in this section. This constraint also means you might not be able to delete a row from the Customers table that has an existing row in the Orders table.

This constraint has several options of its own to dictate what should happen when T-SQL tries to delete or update a row that's tied to another table with a PK/FK relationship. These options are:

  • NO ACTION: (The default.) The operation that would violate the PK/FK relational integrity.

  • CASCADE: This tells SQL Server to delete all rows in the FK table that point to the deleted or changed row. This means if you use this option, when you delete a Customer, all of the rows in the Orders table for this customer are also dropped. Ah, be careful when you choose this option. It make it very easy to delete entire regions of your database.

  • SET NULL: Instead of deleting the FK rows, the FK/PK column value is simply set to NULL.

  • SET DEFAULT: Instead of deleting the FK rows, the FK/PK column value is set to the defined Default value.

Each of these constraints can be established using the SQL Server Management Studio IDE. While the dialog to set constraints is carefully hidden, if you open a connection to the target server and open the Object Explorer, drill into the list of Tables, choose a specific table, right-click and choose "Modify", you'll see a two-part dialog that can be used to modify the table. To access the table's Properties, right-click on the white-space next to the column definitions (at the top). This opens a dialog that supports setting the constraints, as shown in Figure 2.64.

Figure 2.64. Opening the constraints selector dialog.

[View full size image]

This dialog is your gateway to the UI dialogs used to add and delete table columns; set or remove the primary key (and set the PRIMARY KEY constraint); maintain PK/FK relationships; maintain table, FullText, and XML indexes; as well as maintain Check Constraints. For example, to create the CHECK constraint defined earlier on the Discount column, you could click on "Check Constraints..." and fill in the dialog as shown in Figure 2.65. Notice that the syntax for the constraint expression is a bit different. In this case, simply refer to the column to be tested by name. In the constraint I created, I'm still validating the value to be between 0 and 99.

Figure 2.65. Creating a CHECK constraint with the SQL Server Management Studio.

[View full size image]


Managing Databases and Queries with Batches and Scripts

A T-SQL query can contain 0 to almost any number of SELECT statements (within reason). A set of T-SQL statement that can be executed sequentially is called a "batch". These statements can be SELECT statements, local variable declaration, logic statementsany T-SQL. Building your own batches is easysimply concatenate the T-SQL statements together. No, you don't need to add a semicolon between the statements unless it makes you happybut you do need to be aware that there are several rules that dictate which commands can be combined into a script. The entire batch is compiled into a single execution plan that might positively or negatively affect performance.

For example, the T-SQL statement in Figure 2.66 returns two resultsetseach containing a single rowset generated by the two SELECT statements.

Figure 2.66. Returning multiple resultsets from a single T-SQL batch.

[View full size image]

Once a resultset has been (completely) processed, ADO.NET can ask SQL Server if there are more resultsets and the process can be repeated.

Scripts Versus Batches

In some cases, one finds the need to create a file that contains a set of batches to perform a series of T-SQL operationsthis is called a "script". DBAs often use scripts to create entire databases or orchestrate complex changes to the schema.

When creating a script, each batch is separated from the next with a "break" operator (usually "GO" on a separate line). T-SQL is not capable of executing batches on its own, so you'll have to use one of the tools such as SQLCMD, OSQL, ISQL, SQL Server Management Studio, or SQL Server Management Studio Express. See "GO" in BOL for more details.

As I'll show in a minute, if the T-SQL in your batch is too complex, SQL Server can spend a lot of time trying to compile and execute it. A T-SQL batch can also contain other commands. It can contain "action" statements (as I'll discuss in a minute) that make changes to the data or commands that control utilities like DBCC[21] on the server. It can also contain stored procedures to be executed on the server. As I discuss in a few pages, these are simply pre-compiled T-SQL programs you've written ahead of time to improve performance. A SELECT statement that passes a syntax check rarely fails, but if it does, it's usually because you don't have rights to examine a specific part of the data you've asked to retrieve.

[21] DBCC is a utility that can be called from a T-SQL query. It's used to perform a wide variety of maintenance and diagnostic functions on the server.


The Query Optimizer and the Query Plan

Someone could write a whole book on how the SQL Server Query Optimizer (QO) generates a query plan, but there isn't the space here so I'll just hit the high pointsthe rest would probably put you to sleep anyway. The query plan is simply the program (the series of steps) SQL Server runs to perform the work specified by the T-SQL query. The query plan is generated automatically the first time SQL Server encounters the query and does not find a cached plan left over from the last time the query was executed.

It's important to know how the QO builds the query plan, so I'll provide enough information here to help you write T-SQL that the QO can use to build an efficient query planone that works each time the query is executedregardless of the parameters passed with the query. Consider that T-SQL is a programming language like any other. Just like any other computer language compiler, SQL Server must convert your T-SQL query string into a number of low-level operations to locate, sift, filter, and sort the data you ask for. These basic operations include:

  • Table scans, where every single row in the table is read as SQL Server searches or collects the data. These are very expensive and, if your table is large, extremely expensive.

  • Index seeks, where SQL Server uses an available index to locate specific rows to include in the resultset. These can be far faster but might require a lot of disk head movement, as the location of the rows might not be in physical sequential order.

  • Clustered index seeks, where SQL Server uses the table's clustered index to locate rows. This is one of the fastest ways to return data, as a clustered index puts the table's data in physical sequential order as data is added to the database or when the index is rebuilt.

  • Nested loops, which combine data from two or more intermediate sources (see Figure 2.72).

Viewing and Tuning the Query Plan

The SQL Server Management Studio query editor can also be used to show the query's execution plan. I'm going to use the query I built for Figure 2.27. While it's pretty simplistic, if you click on the "Display Estimate Query Plan" button, you'll see a graphical layout of the execution plan (as shown in the Execution Plan tabbed window in Figure 2.67). This is a great way to forecast how well your query will run. As you make changes to the TSQL, you can watch the plan adapt to these changes.

Figure 2.67. A simple execution query plan.

[View full size image]

Note that the first step in the plan is a Clustered Index Scan (the step farthest to the right), which takes about 40% of the time to execute. The two Compute Scalar steps take about 11%, but the sort takes about 49%. Perhaps we could make this query faster by removing the ORDER BY clause and depending on a client-side sortin other words, let the UI element (like a DataGridView control) sort the returned rowset.

You can also choose to show the actual query planthis is a new feature for SQL Server 2005. The actual plan is based on the parameters passed to the query at execution timewhich can materially affect the query strategy generated by the server. We use the estimated and actual query plans to help design stored procedures and other queries that have to perform well no matter what parameters are passed to them.

Another point to note is that the same query executed in SQL Server 2000 builds a different query plan (as shown in Figure 2.68). Notice how the sort is done before executing the scalar and how much more time (relative to the other parts of the plan) is consumed by the Clustered Index scan (the first step).

Figure 2.68. SQL Server 2000 Query Analyzer query plan estimate when executing the same query.

[View full size image]

Examining Client Statistics and Alternate Plans

Another way to better understand what the server is doing when it's executing the query is to study the client statistics. This feature was in SQL Server 2000 Query Analyzer, but you had to add a few lines of T-SQL[22] to your query to enable them. The new SQL Server Management Studio is far better. Consider the simple query I just executed. If I remove the ORDER BY clause and reexecute the query with the "show client statistics" option enabled, I'm presented with a new tab "Client Statistics", shown in Figure 2.69.

[22] SET STATISTICS IO ON and other similar commands were used to show these runtime statistics in SQL Server 2000.

Figure 2.69. Client Statistics Tab showing multiple "trials".

[View full size image]

Notice that there are two "trials" shown. Each time you execute a query when the Show Client Statistics button is depressed, SQL Server records the statistics and compares these results with the previous trial. It also shows (using arrows) where the results did not change or were lower or higher. Notice that in trial 2 (where we removed the ORDER BY clause), the total execution time dropped by 25% (from 50 to 40), and the amount of client processing time dropped from 30 to 0. It does not take a rocket scientist to tell this analysis can be very valuable as you try to determine how to best tune your query or stored procedure.

Management Studio also supports the Database Tuning advisor that can analyze how SQL Server is accessing your database and whether or not your indexes are being used efficiently. It can actually make the corrections for you, if necessary.

Using the Query Options | General tab (shown in Figure 2.70), you can also set the default rowcount (we recommend you set this to about 50), a smaller TEXTSIZE (10K should be plenty), and a finite execution time-out (90 seconds should be more than enough).

Figure 2.70. Setting Query Options on the Execution | General options page.

[View full size image]

Another helpful feature is the way that SQL Server Management Studio has exposed its Query Options page. By selecting the Execution | Advanced options page (shown in Figure 2.71), you can choose a wealth of options that are implemented for each query you executewithout having to add T-SQL.

Figure 2.71. The SQL Server Management Studio Query Options page showing the Execution | Advanced options.

[View full size image]

You can see the query plan built by SQL Server by using Management Studio against virtually any query. Here is a query plan for a query that's a bit more complex, as shown in Figure 2.72. In this case, I've opened a query window in SQL Server Management Studio, entered a T-SQL JOIN query and clicked the Display Estimated Query Plan option button on the SQL Editor toolbar.

Figure 2.72. A query plan as displayed by "Display Estimated Query Plan".

[View full size image]

If you hover the cursor over one of the query operations, Management Studio exposes a detail dialog that shows what low-level operations were done by the chosen operation. Note the thickness of the arrowsthicker arrows mean proportionately more data must move over this path while processing the plan. As you hover over these arrows, you're informed of the exact number of I/O operations performed at that stage.

In the process of building the query plan, SQL Server's QO takes into account any existing table indexes that could help locate the data rows requested or to sort the data at one phase or another of the plan. The QO also uses database "statistics" to determine the best query plan. Statistics are sophisticated maps of the data that SQL Server automatically maintains behind the scenes. Using these statistics, the QO can determine how the data is distributed in the table. These help determine whether a particular query would be more efficiently executed using one or more indexes (an Index Scan or Clustered Index Scan) or by simply scanning each row in the table (a Table Scan) as it looks for rows that qualify for your query. Yes, a query can actually execute more quickly by stepping through each row in a large table instead of walking through one or more indexes to access the rows. For example, if your WHERE clause is missing or so ambiguous to encompass most of the table, it makes more sense to simply process all of the rows in the table. Index operations require two separate stepsone to locate the appropriate index (page) and another to fetch the data page the index addresses.

SQL Server stores its data on 8K "pages" that contain as many rows as will fit. Pages are stored in "extents" and a database can have many, many extents. When SQL Server accesses the data, it usually loads an entire page in memory at once from disk to save time.

Another important point here is that the QO compiles the entire T-SQL query when it builds a query planit does not "shortcut" the plan based on in the T-SQL logic. The QO does, however, factor in the values of parameters passed into the query, so in some cases, two similar T-SQL queries might return very different query plans (and perform very differently) when you use different parameters or use different arguments in the query.

As your T-SQL queries become more complex, the QO has to take more and more time trying to figure out a viable query plan the first time an uncached query is encountered. After a while, it simply gives up and uses brute force to execute the query. You don't want this to happen. When you graduate to more sophisticated T-SQL queries, you can provide "hints" to the QO to "encourage" it to use one type of query plan strategy over another. When you're ready to do that, I can point you to a few books that discuss this in more depth. I also talk about writing efficient queries throughout the bookespecially when I start discussing writing and executing stored procedures.

IMHO

Think more, smaller, not fewer, and larger when creating stored procedures.

Note that SQL Server 2005 introduces statement-level recompilation of stored procedures. When SQL Server 2005 recompiles stored procedures, only the statement that caused the recompilation is compiled rather than the entire procedure. As a result, SQL Server uses the parameter values as they exist in the recompiled statement when regenerating a query plan. These values may differ from those that were originally passed into the procedure.

Executing the Query Plan

Once SQL Server has built a query plan, it's executed by one or more "worker threads" on the server. This means that several threads of execution can work on your query at oncein parallel. As the first few rows (the first few kilobytes of data) are returned from the query plan's program, they're held in a buffer on the server, and the client is notified that SQL Server is ready to transmit the results of the query. SQL Server waits until it receives notification that the client has begun to accept the processed rows. As the client (your ADO code) starts fetching the rows, the server tries to keep ahead by finding more rows to return to the client and buffering them on the server. When the last row is found by SQL Server and fetched by ADO, a final TDS packet is built that contains the rows affected count, the RETURN value, and any OUTPUT parameters (if you're executing a stored procedure). The rows affected value reflects how many rows were found by the query or were changed by an action query (like an INSERT, UPDATE, or DELETE). Nope, you can't get to the rows affected count or the other returned parameters before the rows are sentthat's because SQL Server does not know the final count until it finishes finding all qualifying rows.

It's also possible (but dangerous) to ask SQL Server to "cancel" an in-process operation. If you choose to do this, SQL Server attempts to stop what it's doing, but it won't do anything that it thinks will compromise the database integrity. In some cases, this might mean that one or more operations are "undone" or rolled back.

When the query plan has been completed, the connection is reset by SQL Server and you can execute another query. At least, that's how SQL Server has worked since its inception. SQL Server 2005 now permits the same connection to be used for more than one query at the same time. This technology is called "Multiple Active ResultSets," or MARS. I'll talk about the limitations and rules when using MARS in Chapter 9.


Understanding the Buffer Cache

Fundamentally, the SQL Server buffer cache is a (large) area of RAM that's used as a scratchpada place to store frequently accessed data pages, procedures, query plans, and other detritus needed by the server. In reality, SQL Server uses several buffer caches to hold recently used data and manages them independently. As SQL Server is asked to fetch data from the database, it first looks in the buffer cache(s) to see if the data has been recently been fetched. Since database changes are also written to the cache before being committed to the hard disk, the data pages that are used most often tend to stay in the cache, where they can be retrieved more quickly. The side-effect of this behavior is that SQL Server can actually get faster as more users are added, as long as they're doing about the same thing with the same data, because SQL Server does not have to fetch the data from diskit's already in memory.

IMHO

More RAM makes SQL Server faster.

Managing Cached Query Plans

When SQL Server builds a query plan, it also writes it to the cache, where it remains until forced out by other, (hopefully) more important data or by request. When a query is executed that (closely) resembles a cached plan, SQL Server does not bother to build a new query planit simply uses the plan already in the cache. Generally, that's what happens when SQL Server is asked to execute a stored procedure, but it can also reuse the query plan of an ad hoc query. First, SQL Server checks to see if the stored procedure query plan is already in memoryif it is, it's simply reused. Frankly, it's very likely that SQL Server will reuse a stored procedure query plan (a virtual certainty)[23]. However, the chances aren't as good that an ad hoc query plan can be reused.

[23] One of the problems I'll talk about in Chapter 11, "Executing SQLCommand Objects," is why it's not great that these cached plans are always used. Sometimes, they aren't a good match for what needs to be done and need to be rebuilt.

IMHO

Stored procedure query plans are almost always executed from the cacheeven if they don't really match the query being executed.

Eventually, the cache fills with "pertinent" data rows and query plans. I say "pertinent" because these data rows and plans are the ones that have been accessed the most frequentlythey are the rows and plans that pertain to current client operations. When the cache is full, SQL Server asks for more memory from the OS. If no additional memory is available, SQL Server has to decide where to put newly arriving data and plans. To deal with this issue, SQL Server uses a least-frequently-used (LFU) system to discard items in the cache. To help decide what goes and what gets kept, each time a row or plan is accessed, its "use count" is incremented. All SQL Server has to do is find the blocks of memory with the lowest number to overlay. Sure, when new rows are fetched or when you fetch a Binary large object (BLOB, a picture or large text file stored in the database), the cache can be stressed and some pertinent data and plans might be overlaid to make room for the new data values.

The size of the cache is determined automatically by SQL Server. Its space is allocated out of the memory reserved for the server. You can set the maximum and minimum server memory using the Server Properties dialog exposed in SQL Server Management Studio (as shown in Figure 2.73).

Figure 2.73. Configuring the memory allocation in SQL Server Management Studio.

Generally, one of the easiest ways to improve SQL Server's performance is to increase the amount of memory allocated to SQL Server, so installing more RAM in your server can really pay off in big performance gainsassuming you aren't squandering the new RAM by running a print or report server or playing DVD movies on the same system. However, when working with SQL Server Express edition, your user might not appreciate SQL Server consuming more than its share of memory.

Of course, memory allocation varies quite a bit when SQL Server is used on a dedicated systemthat is, when it's the only software running on the box. In this case, the only competition SQL Server has to contend with is the operating system. This means you can run SQL Server on a smaller systemone that does not need as much power, as it's not distracted by user interaction and other programs. In a small office, I often recommend using that older system that's sitting around collecting dust to host SQL Server. It has plenty of power to do the job.

When SQL Server is installed alongside Microsoft Office and the plethora of other spam filters, virus protectors, VOIP applications, IM, and other stuff many folks run on their work systems[24], SQL Server has plenty of competition for RAM. Depending on SQL Server's role, you might consider setting the maximum server memory to a much lower number. Consider that SQL Server is a "good citizen" when it comes to the memory it uses on the system. It asks for more memory only when it's needed and releases it when requested to do so by the operating system. Other applications are not so benevolent. For example, Microsoft PowerPoint grabs all of the memory on the system (and every system within 40 feetor so it seems) and holds it until the application ends. If you must run an application like PowerPoint on the same system as SQL Server, it's a good idea to set the minimum server memory (as shown in Figure 2.73).

[24] Of course, home systems are even more loaded with background tasks and stuff that can hog CPU and RAM resources.


Executing Action Commands

ADO.NET (as I'll discuss in depth in Chapter 8, "Getting Started with ADO.NET," and beyond) uses an entirely different approach to handling changes to the database. In earlier data access interfaces like DAO, RDO, and ADOc, the action commands were automatically generated based on the SELECT statement used to fetch the rowset. Microsoft found this to be too inflexible to deal with the myriad data sources and ways to return rows. In contrast, ADO.NET leaves creation of the action commands up to you. That is, when it comes time to make changes to the database, your application (or one of the tools like SQL Server Management Studio, SQL Express Manager, or the Server Explorer in Visual Studio) needs to construct one of three types of T-SQL statements:

  • An INSERT statement to add new rows

  • An UPDATE statement to change an existing row

  • A DELETE statement to remove an existing row

The first two of these T-SQL statements use a specific syntax that specifies the database table and the columns to add or change. The syntax in the UPDATE and DELETE statements should use a WHERE clause that tells SQL Server how to locate the row(s) to change or remove (without that WHERE clause, all rows would be changed or removed).

Yes, there are a variety of ways to construct action commands, and the Visual Studio wizards (like the DataAdapter Configuration wizard [DACW]) or CommandBuilder can do it for you. You need to understand the mechanics of these action commands to understand the code being generated and to know how to modify it to better address your situation. That's because the code generated by the CommandBuilder and the DACW (which uses the CommandBuilder) is, shall I say, "challenged." While I might use the Visual Studiogenerated code for simple operations, it breaks down once the SELECT query gets too complex. I discuss these issues in depth in Chapter 6, "Building Data Sources, DataSets and TableAdapters." Let's take a quick look at each of these T-SQL action commands to see what they do and how you can control their behavior.

Inserting New Rows

The T-SQL INSERT statement is used to add one or more rows to an existing table. However, before you run off and start writing routines that generate INSERT statements en masse, make sure you check out SQL Server's ability to import data for you. These "bulk" techniques are dramatically faster than any INSERT scheme you can create. Even if the data has to be munged before it's committed to production tables, it's still faster to upload it to a SQL Server work table (or a #Temp table) and run server-side code to move it to your "real" tables.

In the process of executing the INSERT statement to add the new row, SQL Server performs the following tasks:

  • Addresses the target table based on the ownership chain. If the table is referenced explicitly, this process is clearotherwise, SQL Server uses the "current catalog" or default database to determine which database to change. It also assumes the current login credentials are to be used to reference the schema (owner) of the table.

  • Verifies that the schema or user has rights to make the changes. These rights can span the entire table or apply to specific columns.

  • Substitutes the defined DEFAULT value for any columns not provided, or throws an error if required columns are not provided. Yes, it's possible to define a column to accept NULL values. NULL can be specified either explicitly or by leaving out the column in the INSERT statement (assuming there is no DEFAULT declared for the column).

  • Verifies that the data passed as VALUE arguments to the INSERT statement is the correct datatype. Basically, this means SQL Server checks to see that if the column is defined as a number, the VALUE argument is actually a number, or if it's a string, the VALUE provided is a string, and so forth. As far as normal string data types (char, Nchar, varchar, Nvarchar), the server simply truncates over-length strings to match the capacity of the column. ADO.NET, on the other hand (and some of the tools), throws an exception if the string length provided is longer than the declared maximum length.

  • Verifies that the data passed does not violate any rule or check constraint defined for the column. I discuss rules and check constraints later in this chapter.

  • If a column has been defined as IDENTITY, SQL Server generates a unique integer value (one higher than the highest IDENTITY value in the table) and assigns it to the value of the identity column. Likewise, if a column has a default set with the NEWID function, SQL Server generates a new GUID if you don't provide a value.

  • If a primary key (PK) is defined for the table, SQL Server verifies that a row using this PK does not already exist. If the PK already exists and a unique constraint is defined, the INSERT is rejected.

  • If a relationship has been defined for the table, SQL Server verifies that the parent table contains a row that corresponds to this new row (assuming that the new row is child).

  • The row is added to the internal (RAM-based) INSERTED table.

  • If an INSERT Trigger has been defined for the table, this code is executed. I introduce triggers later in this chapter. Again, this code can reject or accept the INSERT.

  • Records the operation in the transaction log. In the background, SQL Server then adds the row to the specified table.

  • If a transaction has not been started, SQL Server adds the row to the appropriate data table page and, if necessary, allocates another page or extent to expand the table. If a transaction is open, the row is added logically (and cached) but not recorded to the data table data page until the transaction is committed.

  • SQL Server updates any indexes defined for the table.

  • SQL Server then returns the RowsAffected value to the program to indicate how many rows were affected (added).

Sure, it's possible to code an INSERT statement to add several rows at a time. For example, to "import" rows from another table, you could build an INSERT statement, as shown in Figure 2.74.

Figure 2.74. Adding rows to a table from another table.

In this example, the target table (TestInsert) is defined to include an IDENTITY column. This means I don't (shouldn't) have to specify the IDENTITY column value in the INSERT statement.

Handling NULL and DEFAULT Values

When you need to add a new row but the value for a specific column is unknown (as in "DateOfDeath" or "FinalVoteCount"), you should provide NULL as the value to be inserted into a specific column. In order for this to work, you must define the table to accept NULL for the column or don't include the column in the column list in the INSERT statement. You can define default values in T-SQL, and these can be assigned to specific table columns. The rule is, if a default is defined for a column and no value is passed in the INSERT statement, the default value is used. If you pass a NULL as the column value in the INSERT, NULL is usednot the defined default.

Don't use the INSERT command to move data from one data source to anotherat least limit the number of rows added to a few dozen rows. ADO.NET provides a bulk copy interface (BCP) to do this more efficiently.

If the INSERT statement succeeds, SQL Server returns the number of rows affected. In the example in Figure 2.74, two rows were added to the TestInsert table. This value is passed back to ADO.NET, and if you set up an argument to capture it, you can inspect this value. This can be very handy when you want to know if the INSERT did what you asked/expected it to do. ADO.NET also expects this behavior. When I discuss the DataAdapter Update method in Chapter 12, "Managing Updates," I'll show that ADO.NET uses the rows affected value to determine if the INSERT, UPDATE, or DELETE operation succeeded. In this case, ADO.NET always expects a rows affected value of 1.

The rows affected value is returned to ADO.NET after each action statement in the batch.

Capturing the New Identity Value

Many of the tables I work with use Identity values or GUIDs as the primary key. When I add a new row, it's important to know the new SQL Servergenerated value, so I can manage parent/child relationships in the client. Retrieving the new value can be done in a variety of ways:

  • Fetch the entire row just after the INSERT statement (in the same batch, as shown in Figure 2.75). In this case, you would add a T-SQL SELECT to the batch being executed that returns the newly inserted row by using a WHERE clause that references this new row by using the SCOPE_IDENTITY() function. If the new GUID is generated by the NEWID function, you can fetch it with a SELECT.

    Figure 2.75. Executing a SELECT directly after the INSERT to retrieve the Identity value.

  • If you need the newly generated Identity value, fetch it with the SCOPE_IDENTITY() function. Figure 2.76 illustrates T-SQL code that retrieves the newly created Identity value based on the current code scope for the current connection. I prefer to use this function over IDENT_CURRENT() or @@IDENTITY for most situations, as it eliminates issues caused by other operations adding rows.

    Figure 2.76. Fetching the newly generated Identity value directly after an INSERT.

  • Use the IDENT_CURRENT() function; you can use a SELECT statement to return the last Identity value generated for a specific table. This value is updated when any rows are added to the table by any application in any scope, so it's important to fetch this value directly after the INSERT has been executed (as shown in Figure 2.77).

    Figure 2.77. Using a SELECT from SCOPE_IDENTITY to fetch the Identity value.

  • Use the @@IDENTITY global variable. In this case, you can execute a query (in the same batch) to return the last identity value generated by SQL Server. I don't recommend this approach, as @@IDENTITY can be reset by a trigger launched by the INSERT that adds a row to another table that contains an IDENTITY column.

The Visual Studio 2003 DataAdapter Configuration Wizard generates an INSERT statement that can include an additional SELECT statement directly after the INSERT that uses the @@IDENTITY global variable. After I suggested that Microsoft change this to SCOPE_IDENTITY(), they did so. Visual Studio 2005 now does it right. When I get to Chapter 12, I'll show you how to create client-side Identity values to manage parent/child relationships before and after ADO.NET executes the DataAdapter Update method.

Inserting Data with SELECT INTO

I've found that it's often handy to insert data into a table using SELECT INTO syntax. Suppose you need a temporary table that contains information you expect to use in subsequent queries. If you use SELECT INTO, you can create a permanent or temporary (#Temp) table on-the-fly. This could not be much easier. Write normal T-SQL SELECT, but add the INTO <tablename> before the FROM clause, as shown in Figure 2.78.

Figure 2.78. Using SELECT INTO to populate a #Temp table.

[View full size image]

If you prefix the target table name with "#", the new table is created in TempDB. Yes, you'll have to drop this table yourself once you're done with it because #Temp tables live for the life of the connection and are not visible to other connectionsnot even other connections your application opens.

Figure 2.79. Using SELECT INTO to insert new rows into a temporary table.

[View full size image]

Using Temporary (#Temp) Tables

#Temp tables can be very handy when trying to persist a set of rows like a notepad. These tables are stored automatically on a connection-by-connection basis in TempDB. When the connection is dropped, any residual tables are dropped. TempDB is re-created each time the system boots (it's copied from the model database), so there can never be anything left over from a previous existence. A few points to remember:

  • #Temp tables can be created by using SELECT INTO, as I just described. If you create them in an application with a persistent connection, the #Temp table remains in place as long as the connection remains open and you don't explicitly drop the table.

  • #Temp tables created by a stored procedure are "owned" by the stored procedure and are dropped once the procedure is done. Sure, you can create a #Temp table, and other stored procedures called from the creating procedure can see them. Once the stored procedure that creates the #Temp table ends, it is dropped.

  • #Temp tables consume resources in TempDB. If you plan to use this approach to build large #Temp tables or quite a few of them, you'll want to stretch the size of TempDB ahead of time. Otherwise, your system start off more slowly as TempDB is incrementally stretched to accommodate more rowsets. To change the initial size of TempDB, use SQL Server Management Studio tempdb property page to reset it.

Figure 2.80. Changing the initial size of TempDB.

[View full size image]

You can use #Temp tables like any other table. This means you can include a #Temp table in a JOIN to improve query performance in situations where a connected application makes a number of similar queries.

Updating Rows

Yes, I expect that at some time or another, you're going to want to add, change, or delete the data in your databaseunless it's on CD. This section walks you through the process of making those changes using T-SQL. Sure, the Visual Studio wizards will help build UPDATE, INSERT, and DELETE statements for you, but as I discuss in Chapter 12, you'll quickly outgrow the approach they use.

Changing SQL Server data is not that hard to do if you remember three basic tips:

1. | Don't change the primary key.
2. | Don't change the primary key.
3. | See tip 1.

When it comes time to change your database, most of the work involves finding the row that needs changing, setting the new column values, and dealing with concurrency issues. If your UPDATE statement does not have a WHERE clause that identifies a single row, more than one row can (and probably will) be changed. As you'll see in Chapter 12, ADO.NET expects you to write UPDATE commands that address rows individually, so updates are done one row at a time.

Visual Studio and ADO.NET can generate appropriate T-SQL UPDATE statements for youalthough as your application grows in sophistication, you probably won't want them to do so. The other major aspect of updating involves concurrency, which I discuss next, and how ADO.NET handles concurrency, which I discuss in Chapter 12.

Your T-SQL UPDATE should not attempt to update the row's primary key. If you think about it, when you change the primary key (the value[s] that define the "uniqueness" of a row), you're really dropping an existing row and replacing it with another. If you keep this in mind, your code can be a lot simpler and ADO.NET won't be as petulant.

The T-SQL UPDATE statement is used to change one or more rows to an existing table. In the process of changing the row, SQL Server performs the following tasks for each row addressed by the WHERE clause:

  • Addresses the target table based on the ownership chain, just as was done for the INSERT statement.

  • Verifies that the schema or user has rights to make the changes. These rights can span the entire table or apply to specific columns.

  • If the SET statement specifies that the DEFAULT value is to be used, fetches it.

  • Verifies that the values passed as expression arguments to the SET statement are the correct datatypes, as was done for the INSERT statement, and that the columns specified are correct. Ensures that the values conform to the datatype constraints.

  • Verifies that the data passed does not violate any rule or check constraint defined for the column. I discuss rules and check constraints later in this chapter.

  • If the UPDATE changes the primary key (PK), the system has to treat the UPDATE as a DELETE followed by an INSERT. In this case, if a primary key (PK) is defined for the table, SQL Server verifies that a row using the updated PK does not already exist. If it does and a UNIQUE constraint is defined, the UPDATE is rejected. In addition, if a relationship has been defined for the table, SQL Server verifies that that the parent table contains a row that corresponds to this "new" row (assuming the new row is child).

  • The row is added to the internal (RAM-based) INSERTED and DELETED tables. The INSERTED table stores copies of the affected rows during INSERT and UPDATE statements. Note that an UPDATE operation is similar to a DELETE followed by an INSERT. The old rows are copied to the DELETED table first, and then the new rows are copied to the target table and to the INSERTED table.

  • If an UPDATE Trigger has been defined for the table, this code is executed. I introduce triggers later in this chapter. Again, this code can reject or accept the UPDATE.

  • Records the operation in the transaction log. If a transaction has not been started, SQL Server adds the row to the appropriate data table page and, if necessary, allocates another page or extent to expand the table. If a transaction is open, the row is added logically (and cached) but not recorded to the data table data page until the transaction is committed.

  • SQL Server updates any indexes defined for the table based on changes made to any indexed columns.

  • SQL Server then returns the RowsAffected value to the program to indicate how many rows were affected (updated).

Understanding the TSQL UPDATE SET Clause

The UPDATE statement's SET clause contains a set of values to replace values in the specified columns. The values can also contain an expression based on the current contents of the row, as when the new value is to be computed relative to the existing value"CurrentSales + 2.50".

The example shown in Figure 2.81 illustrates using an UPDATE statement to set the SalesPerson table's Bonus column for all salespeople whose sales are within 10% of the top sales amount. Sure, this UPDATE T-SQL operation could have been written in a number of different ways, so it's a great idea to click on the "Include Client Statistics" in SQL Server Management Studio as you build the T-SQL to see which is most efficient, given the table indexes.

Figure 2.81. Using the UPDATE statement to change a specific column in many rows.

[View full size image]

Note that the SET statement is used (in this case) to apply a value to a specific column. It's not unusual to use SET against several columns in a single UPDATE operation, as when your code is updating a row with more current information.

Of course, executing any UPDATE statement assumes that the current User associated with the connection has rights to make changes to the specific table and columns being changed. Imagine (if you will) a SalesPerson table that contains data your User has rights to see but not change. For example, while the User might be able to execute a SELECT against the SalesPerson table, the account might not have rights to alter any but the Bonus columnand then only because specific update rights have been granted to that User account.

Assuming your User has sufficient rights, a more typical UPDATE statement might reference several columns in a single operation, as shown in Figure 2.82.

Figure 2.82. Using the UPDATE SET operator to change the values of several columns.

But no, this approach doesn't make much sense, as it requires you to recode the T-SQL for each change you want to make. In most cases, you'll want to create a parameter-based query to accept the values to SET, as well as the criteria to choose the row(s) to UPDATEit's rarely a good idea to hard-code assigned values in an UPDATE statement.

Parameter-Based Updates

Parameter-based UPDATES are designed to permit your code to pass in values to be used with the SET statement. These can be implemented in your code by:

  • Creating an ADO.NET Command object that contains the T-SQL UPDATE statement and creating a Parameters collection to manage each parameter.

  • Creating a stored procedure that accepts these parameters. In this case, the Command object simply points to the stored procedurethe Parameters collection is still the same.

  • Using Visual Studio to create a DataSource-based TableAdapter to handle the update.

  • Using Visual Studio's DataAdapter Configuration Wizard to build code to handle the update.

All of these approaches are discussed in the context of Visual Studio and ADO.NET in Chapter 6 and again in Chapter 8, as they involve using ADO.NET or Visual Studio to generate the code.

Let's take a look at a simple stored procedure that inserts the parameter values into the UPDATE statement at runtime (see Figure 2.83).

Figure 2.83. A stored procedure used to perform an UPDATE using parameters.

To test this procedure, I need to build a "test harness". A test harness is simply a program used to exercise the procedure. It can be written with Visual Studio and a SQL script, and executed with SQLCor, as in this case, I use SQL Server Management Studio. This T-SQL script (shown in Figure 2.84) sets up a local variable to capture the stored procedure RETURN value.

Figure 2.84. Testing the update stored procedure.

[View full size image]

Since we're working with Visual Studio, we can also test this (or any) stored procedure using the Server Explorer. Sure, you can also use Visual Studio to create, edit, and step through the stored procedures. Later in this chapter, I'll show you howsee "Introducing Stored Procedures."

Strongly Typed Data Columns

When you assign a value to a table column, SQL Server makes sure that the data is correctly "typed." That is, the source data (the information you're trying to apply to the column) is the right size and "shape." For example, if you define a column as tinyint, acceptable values are numbers[25] between 0 and 255. If you provide a value higher (or lower) than the datatype permits, SQL Server will throw an exception. Each datatype has its own constraints, be it a range of values (including the ability to hold negative numbers), precision and scale, or length. Datatypes also determine how the data is stored, as in Boolean, binary, text, string, floating point, or Unicode/ANSI designations.

[25] Remember, everything stored in the database is binary and the tinyint column holds 8 bits, which means it can hold a binary value of 00000000 to 11111111 or 0 to FF in hexadecimal.

If you take a closer look at the example shown in Figure 2.85 and change the second stored procedure parameter to "234567", the code throws a runtime exception because the column (and parameter) are defined as decimal(6,2). In this case, the datatype is set to permit decimal values no longer than six digitstwo of which are after the decimal point. This means a value of 1234.56 is acceptable, but 12345.67 is not.

Figure 2.85. Type-checking data.

[View full size image]

Datatypes and data validity constraints raise an interesting issue: Should your T-SQL code provide datatype filtering to prevent exceptions, should filtering be done by your own code as the data is captured, or should you code server-side Rules and Constraints to ensure that inbound data is correctly formatted? In any case, your application or server-side code needs to deal with this issue one way or anothereither by filtering inbound data at the source or on the server, or by writing exception handlers to clean up the mess caused by a data exception. I like to code our front-end applications to ensure that data values are captured correctly as the data is entered. This ensures that corrupted data does not enter the system in the first place. Generally, the person entering the data is the "source," so they should know the correct value. It's up to your application to make sure they know how to pass that value to your program. If they're entering the data incorrectly, it's as much your fault as it is theirs. Your program should make the acceptable values and constraints of the data being entered perfectly clear.

IMHO

If users are entering data incorrectly, it's as much your fault as it is theirs.

ADO.NET, the .NET Framework, the languages, and Visual Studio all have mechanisms to help you make sure data values meet the datatype criteria. The mechanism they use is called "strongly typed" data. When you build strongly typed DataSets, DataTables, and other ADO.NET data structures, additional code is added to your project to ensure that only correctly formatted data is captured. This code is added automatically to your project in the form of classes that define each data column in great detail. While it's not necessary to use strongly typed data classes, some developers think that they can help code execution performance (they do) and type safety (they do). Type safety simply means that the generated code that manages your data does not permit invalid data to be moved into your ADO.NET DataRow.

Dealing with Strings

When working with strings, your code needs to deal with several other factors:

  • Do you intend to store ANSI (8-bit characters) or Unicode (16-bit characters)? When storing Unicode characters, remember that the defined length (Nvarchar(20) has room for 20 characters, but it consumes twice as many bytes to store.

  • Is the source data string longer than the defined length? If you attempt to set the column to a string longer than the defined length, an exception is thrown.

  • Are you working with long strings that have virtually unlimited size? In this case, you can store strings up to about 2GB in length (if that makes sense), but performance can be materially impacted.

Dealing with Decimal and Floating Point Numbers

When you need to store numerical money or scientific data in the database, you must choose an appropriate datatype to floating the values on either side of the decimal point. The column datatype specifies how much and the precision of the data to be stored so your code will have to deal with inbound data to ensure it meets these criteria. If you have to store money values, keep in mind that some currencies have very low exchange rates. This means that if you have to store a money value in Venezuelan bolivars or Japanese yen, your money type might have to be capable of handling much larger values when compared to euros or U.S. dollars. For example, $10,000 can be exchanged for about 21,446,000 bolivars. This means in order to save this value, your column needs at least three more digits of scale. However, when you consider that you might be asked to store a value of 2.50 bolivars, your column will need to be able to store $ 0.001165 or round off to the penny.

Dealing with BLOBs

Binary large objects (BLOBs) are simply data that won't (until now) fit in a conventional varbinary or varchar column, like pictures of your Aunt Tillie on her Harley or those contracts you save with your customer data. In SQL Server 2000, if you had a picture or text string that was longer than about 8,000 bytes, you had to store it in a TEXT or IMAGE datatype. In this case, ADO and ADO.NET had to perform a number of tricks to get the data in and out of the database. In SQL Server 2005, BLOB data can now be stored in "ordinary" varchar, nvarchar, varbinary, and XML columns, and ADO.NET does not have to perform as much magic behind the scenes.

In SQL Server 2005, Microsoft has made a number of changes to help manage "large" and super-large data elements easier. First, they added varchar(max) and nvarchar(max), which accept variable-length character and Unicode values as large as 2GB (which means about two billion ANSI characters and about a billion Unicode characters). This means you might not need to use the TEXT datatype at all to store long text strings (like entire documents).

But if you choose to store BLOBs in the database, that's just the beginning of your problems. To start with, getting a BLOB saved or retrieved from the database is far slowerup to six times slower than reading or writing a file containing the same information. Sure, SQL Server 2005 handles BLOBs faster than ever, but it's still slower than simply saving the data to a file and recording the path to the file in the database. When you save a BLOB to the database and you don't plan ahead, you'll end up backing up the BLOB data along with all of the other data, thus slowing down the backup/restore process. Sure, you can save BLOBs to tables on different database segments and back these up separately, but that's a bit tough for those shops without an experienced DBA. The question is, why back up the BLOBs at all if they are static pictures or documents? Yes, if the data is constantly changing, it's a great idea to keep backups and archives. However, if the data is RO, there is no reason at all to keep more than a single backup.

Once you fetch a BLOB into your application, it's addressed via a DataRow column. Can you point Microsoft Word or PowerPoint or a picture editor at this column? Hardly. Generally, you'll find it's a lot easier (and faster) to save the BLOB to a file and point the external application at this file. The application (like Word) can save the file (to disk) using the normal Save clicks, and your application can persist this file to the database. But again I ask, why? Why not simply store the file path in the database and keep the files on normal disk directories? RO files can also be shipped with the application on RO media like CDs or DVDs.

Designing in Concurrency

The next big challenge you'll have to think about as you add, change, and delete rows is concurrency. That is, what has happened to the data since your application last read it? Concurrency is a complex subject, and Microsoft spends quite a bit of time discussing it in its documentation and training materials.

If traffic engineers thought about highway intersections the way that programmers think about data collisions, they would spend all of their time worrying about where to place the ambulances instead of how to time the traffic lights and design the intersections to avoid collisions in the first place.

In a single-user application, you'll have very little need to add code to deal with concurrency. However, if you even suspect that your application will have to scale to support two or more users accessing the same data, you'll need to add code to deal with concurrency contingencies:

  • Should your application be designed to prevent changes to rows while they are being edited? If so, what mechanism should be used to release the locks once the changes are made? Should humans be involved in this locking mechanism? That is, should your application depend on the user to release the lock in a timely manner or make decisions regarding data consistency?

  • What role should transactions play when making changes to the database?

  • When changes are made to the database and the relational schema is complex, one often has to deal with concurrency issues spanning several tables. If a change cannot be made to one of these related tables, what mechanism is used to ensure data and referential integrity?

  • As you add rows, does the data row you're trying to add already exist? Do the current values match the values you are trying to add? How should the differences be reconciled? Should the user do the reconciliation, or should your application do it?

  • As you update a row, has the data changed since it was last read? If so, are the changed columns relevant? That is, you might have read a row but don't really care about some of the columnsthey're irrelevant to the current application.

  • Is the row you're updating still in the database? Has another user (or your own application) deleted it? Should you replace it?

  • Should users be permitted to change the primary key? If so, they aren't updating the row, they're essentially deleting it and adding a new row.

  • Does your user "own" the data? I often separate the database into "regions" that are owned by a specific class of user and prevent other users from updating the data. This avoids a number of situations that ultimately cause collisions.

  • Should the user be involved in the decision about which version of the data is to be saved?

  • Should a changed row be updated? If the row was already changed when your code tried to update it, should your updates also be applied, should the current row be left intact, or should the row be reverted to another state? How should the user be informed of this situation?

As you update your data, Visual Studio can help manage the low-level operations to detect if a collision has occurred. In addition, changes in Visual Studio 2005 make it easier to autogenerate appropriate code for many of the techniques I use to detect collisions.

Let's discuss techniques you can use to address some of the issues I just brought up.

How Can Your Code Determine Whether the Row Is There to Update?

Your T-SQL UPDATE statement's WHERE clause should specifically address the row(s) to change. This is usually implemented by referring to a specific primary key value directly, as shown in Figure 2.82, or via parameter, as shown in Figure 2.83. Sure, your WHERE clause can refer to several columns, if that helps further narrow the focus of the UPDATE. If this WHERE clause returns no rows, that's a pretty good indication that the row is not present to update. No, it does not help to execute a SELECT before executing the UPDATE to determine whether the row is presentunless these are bound into a single atomic transaction. That's because by the time your code returns to post the UPDATE, the row might have been deleted.

In the past, I had to add code to check for duplicates before adding new rows. That's no longer needed with SQL Server as if you define a unique primary key for the table. This is done (as I've said before) by selecting one or more columns that, when taken together, provide a unique way to identify the row. If a unique PK is defined, as you add a new row, SQL Server won't permit you to add a row with the same PK values. This means your table can never contain duplicatesunless you do something really dumb.

If the row you're trying to change is not found, you have to ask yourself why. What happened to it? As often as not, your own application deleted it in another operation. If not that, then the next most likely reason is that the row was deleted by another user or by the DBA performing maintenance. In any case, you need to decide on a fall-back planwhat to do when the row to be updated is not found at all. One approach is to change the UPDATE into an INSERT, but this might be dangerous. You really need to know why the row was removed.

How to Tell Whether the Row Has Changed Since It Was Last Read?

Once your UPDATE statement's WHERE clause has picked the row to change, you need to determine whether the row has changed since it was last read. The easiest way is to use the row's rowversion column valueassuming your row has a rowversion column. A rowversion (or timestamp[26]) column is simply a SQL Servermanaged counter that's automatically incremented as rows are added or changed. When you initially fetch the row, your SELECT should return the current rowversion column value. This way, when it's time to execute the UPDATE, you can compare the server-side rowversion value with the value initially fetched. This test is fast and relatively foolproofit also does not require you to compare other column values against server-side data values that might not have any bearing on your operation. Figure 2.86 illustrates how to set up an UPDATE statement and test harness to use the Timestamp column value to determine whether a concurrency collision has occurred.

[26] Microsoft wants us to use "rowversion" instead of timestamp for better ANSI SQL-2003 compliance.

Figure 2.86. Testing for a concurrency collision using the TimeStamp (rowversion) column value.

[View full size image]

Okay, suppose you permit other users to change the data row while your application is making changes. If you can't use the rowversion or timestamp column, you might need to test for changes in specific columns to determine whether a concurrency collision has occurred. In this case, you'll need to compare pertinent columns against the server-side data.


Introducing Stored Procedures

As a developer, you're likely to hear quite a bit about the wisdom (or lack thereof) of using stored procedures. Some of the naysayers come to the argument with a personal agenda or simply fear of something they don't really understand well enough to endorse. I've been working with SQL Server from the beginning, and I (like 90% of the people I poll in my conference sessions) feel that stored procedures can make the difference between an amateur and a professional application.

Stored procedures are just thatthey're scripts of executable code that are stored on SQL Server for later execution. No, unlike Visual Basic .NET or C# programs, they're not precompiled and salted away somewhere in binary form. Stored procedures (at least, the T-SQL variety) are simply saved to the user or system database, where they can be called up, compiled, and executed when needed. Let's step through the process of creating and executing a stored procedure.

1. | The stored procedure is created in an ordinary text editor (like Visual Notepad), Visual Studio, or SQL Server Management Studio's query editor window. These tools can help author any queries or action commands incorporated in the procedureI'll show you how in Chapter 5.
2. | When "saved" by Visual Studio or executed by SQL Server Management Studio, the CREATE PROCEDURE statement tells SQL Server to record the stored procedure to the database catalogsonce it passes syntax and initial object "sanity" checks.
3. | When the stored procedure is executed, SQL Server retrieves the stored procedure from the database catalogs on disk, compiles the procedure, and makes sure that the objects referenced are accessible, given the rights of the user executing the procedure.
4. | The compiled stored procedure is passed to the query optimizer that, given the input parameters passed by the code executing the procedure, decides how to perform the operations requested. This becomes the query plan (as I discuss later in this chapter).
5. | Using the query plan, SQL Server executes the compiled stored procedure and saves it to the common data/procedure cache. Any resultsets are returned to the calling application.
6. | When the space occupied by the stored procedure in the data/procedure cache is needed, it's overlaid, and the query plan and executable code are lost.

A performance benefit can be achieved when a stored procedure is still in the cache when an application calls it again. In this case, the cached plan and procedure executable are reusedeven if the input parameters used would generate an entirely different query plan. This means that a stored procedure might not run any faster than an ad hoc query (one submitted by your application as raw T-SQL). That's because the process for executing an ad hoc query is basically the same. If there is a query plan and compiled executable in the cache generated by a previous execution of a (very) similar ad hoc query, it's reused.

So why bother with stored procedures? Well, stored procedures move the T-SQL out of your applications and into a central location where they can be managed, tuned, corrected, and protected by the DBA and the entire staff. It means when schemas or business rules change, this shared code can be morphed without having to redeploy your application (at least, part of the time). Consider that a stored procedure can be granted rights to tables that the developer might not (or should not) have rights to access. It can encapsulate operations so that complex update or query operations can be made with ease without having to worry about the underlying details.

Throughout the book, I show how to create, edit, test, and interactively debug stored procedures using Visual Studio's Server Explorer (in Chapter 5) and with ADO.NET (in Chapters 10 12okay, in most chapters). In SQL Server 2005, it's possible to write stored procedures and other server-side executables in Visual Basic .NET or C#. I devote Chapter 13 to this new paradigm.


Introducing Triggers

When implementing a SQL Server database, developers have yet another tool (beyond constraints) to handle a litany of issues having to do with referential and data integrity management, business rules, and many other demands placed on their database systemsTriggers. Basically, a trigger is a stored procedure that's fired automatically. Triggers are associated with database tables and come in two flavors in SQL Server 2005:

  • Data Manipulation Language (DML) triggers. These triggers fire when the INSERT, UPDATE, and DELETE operations are executed against the data table.

  • Data Definition Language (DDL) triggers, which are new for SQL Server 2005. These triggers fire when the data table structure changesnot when the data changes. For example, when the CREATE, ALTER, or DROP statements are executed, corresponding DDL triggers can be created to fire.

Any trigger can execute a ROLLBACK command that stops the operation being requested. This means if the trigger logic discovers that there is something wrong, it can prevent the change from taking place. Let's step through a couple of simple examples.

Creating a DML Trigger

Suppose your company decides that the maximum discount you're going to offer this week is 65%. While some of your applications might offer more, you want to make sure that no new customers are added with a higher discount rate. To accomplish this, you can add a trigger to the Customers table that fires on UPDATE or INSERT operations and that looks like Figure 2.87.

Figure 2.87. DML Trigger to manage maximum discount permitted.

[View full size image]

Notice that the trigger fires on either INSERT or UPDATE operations. Sure, you can create three different triggersone for each action. If the trigger logic determines that the UPDATE or INSERT should not be allowed, it executes a ROLLBACK TRANSACTION to undo the change.

A unique feature of the DML trigger is the ability to access the row(s) being changed via the INSERTED and DELETED pseudo tables. These can be essential when your trigger logic needs to compare two or more columns or generate a new value to place in a columnoverriding the value that would be written to the database. The DELETED pseudo table can be inspected as well. It contains the row(s) that are about to be dropped from the database. As in the movie Robocop, you could roll back a DELETE if the customer being rubbed out is in the list of company executives.

Creating a DDL Trigger

Data Definition Language (DDL) triggers have a much broader scope. Once created, they monitor all indicated changes on the database level or on the entire SQL Server instance. No, this means you can't create DDL triggers for a specific table. In the example shown in Figure 2.88, I create a DDL trigger that prevents any table from being dropped if any customers still owe us money. A bit contrived, but still....

Figure 2.88. DDL trigger to prevent anyone from dropping tables.

[View full size image]


Using Transactions to Protect Data Integrity

Sometimes, you must submit several "related" change operations to a database that should be completed together or not at all. For example, if you're moving money from your savings account to your checking account, you perform operations that debit the savings account and credit the checking account in one "atomic" operation. Atomic operations are those that cannot be broken down into smaller parts and still maintain integrity of the database and its data. In an atomic operation, if one operation succeeds and the other fails, the system "creates" money. This is frowned upon, unless you happen to work in Washington, D.C., where they do this all the time.

SQL Server helps manage atomic operations by tying two or more operations together in a "transaction." If any one of the operations in the transaction fails for any reason, SQL Server automatically executes commands to undo or "roll back" the changes so the database is returned to the same state as before the operations.

SQL Server can also permit you to intentionally roll back changes made in your action command simply by adding appropriate Transaction commands to the T-SQL query. This is one way to experiment with "what if" queries or changes. That is, you can begin a T-SQL query with a BEGIN TRANSACTION to mark the point where subsequent operations are not to be committed (written) to the databaseat least, not until committed. After this point, you can code action commands to change the data and SELECT statements to return the changed data, including summary totals, other aggregates, or expressions based on the changes. Once the first set of operations is complete, you can execute COMMIT TRANSACTION to save these changes to the database or ROLLBACK TRANSACTION to reset the database state.

Clearly, transactions can be expensive, since SQL Server must juggle two data statesone before the transaction began and another afterwhich reflect the changes. Transactions can also lock regions of the database and prevent changes until they are committed or rolled back. It's for this reason that I recommend that you have a firm understanding of how to manage transactions and their side-effects before you start using them with wonton abandon in your T-SQL queries.

Transaction state is managed on a connection basis. That is, if you establish a connection and execute a "BEGIN TRANSACTION", from that point forward, any changes you make to the database are not written to the database, but saved so they can be committed at a later time. If you disconnect (or get disconnected) from the server (as when the new janitor unplugs the server to recharge his MPEG player), the database is left in a limbo state until it reboots. When you discover the SQL Server is down, you "counsel" the janitor about unplugging stuff and reboot the server. During its initialization process, SQL Server checks the transaction log for any uncommitted changes or pending transactions. Since your changes were never committed, the server assumes (correctly) that these changes should be rolled back, and it does so. If the server was partway through the process of writing a batch of "committed" changes to the hard disk when the server went down, SQL Server completes the commit process before returning control of the database to any operations. This way, the database integrity remains intactautomatically.


Administrative Functions

Just like any data store, SQL Server requires maintenance. This is true if the database is managing a Fortune 100 company or your desktop collection of CDs. This means your database must be backed up periodically. Just how often depends on the value of the data, how often it changes, and how often you have access to the database. Without stating the obvious, this is a very serious subject for some companies and not much of an issue for others. Regardless, you should learn how to back up SQL Server databases. SQL Server Management Studio makes this pretty easy.

One easy way to back up a database is to simply "detach" the database (from master), copy the file to a safe location, and "attach" it again. This can be done easily with SQL Server Management Studio but requires that all connections be dropped beforehandor the detach does it for you. Getting into SQL Server Management Studio is easyjust choose it from the Start/Run menu and open a connection to your server instance as SA or using a Windows login that's in the Administrators group. From there, all of the maintenance functions are pretty easy to access.

Attaching a .MDF Database File

To attach any database to master:

1.
Right-click on the Databases icon in the Object Explorer and choose "Attach".

2.
This opens a dialog that prompts you for the path of the database to attach. Click the Add button to open a file finder dialog, as shown in Figure 2.89. Point to the .MDF file to attach.

Figure 2.89. Attaching a database with SQL Server Management Studio.

[View full size image]

3.
Choose an .MDF file from the list. No, the file cannot be on a network share. SQL Server does not permit that. Note that the .MDF file might also have an .LDF (log file) in the same directory (or elsewhere). This is captured automatically. Some databases are stored in several .MDF filesall of which appear in the Attach Databases dialog once you choose a root .MDF, as shown in Figure 2.90.

Figure 2.90. Attaching a multi-file database with its log file.

[View full size image]

4.
Once you've selected the database(s) to attach, click OK. SQL Server takes it from there.

If the .MDF file is from an older version, SQL Server validates it and converts it to SQL Server 2005. In this case, the files will be altered and cannot be accessed by the previous version. Note that you can't attach a database whose name already exists or whose files are already attached to masteron any SQL Server. If you import an older version, the database is marked as being compatible with the old version (not 9.0the SQL Server 2005 version). This means that many of the SQL Server 2005 features and new T-SQL syntax cannot be used against the newly attached databasenot until you set the compatibility level to 9.0 (or to the latest version). This is accomplished by opening the database Properties page (right-click on the database in the Object Explorer and choose "Properties"), navigating to the Options page, and setting the Compatibility level, as shown in Figure 2.91.

Figure 2.91. Setting the database compatibility level and other options.

[View full size image]

Setting the Auto Close Option

While we're on the options page, note that you have the option of setting how the server behaves when the last connection is closed (as when SQL Server Express is deployed to a user's system). In this case, it makes sense to set the Auto Close option to True. When True, the database is shut down cleanly and its resources are freed after the last user logs off. This makes it easy (and safe) to copy, delete, or uninstall the database and the applications that use it.

Importing and Exporting Data

If SQL Server does anything well, it's importing and exporting data. Given years of experience with customers wanting to bring existing data into SQL Server, Microsoft has created a highly respected data import engine to handle virtually any type of data from virtually any source. All of this functionality has been refined and improved over the years. The SQL Server 2005 version now calls it "SSIS" for SQL Server Integration Services. It's the successor to its very popular DTS (Data Transformation Services). I'm not going to bore you with a walk-through, as the SSIS wizard launched from the same SQL Server Management Studio database "Tasks" option menu is very intuitive.

Backing Up SQL Server Databases

SQL Server Management Studio can also run a "real" backup of your database without disconnecting or disrupting anyone. This means you can safely (and fully) back up a live production database with dozens of connections open and active. To do so:

1.
Drill into the Object Explorer in SQL Server Management Studio, select the database to back up, and right-click. At this point, you can choose to detach or back up the selected database from the Tasks list. In this case, we're going to back it up. If you choose to detach an .MDF, drill down to the physical file location so you'll be able to copy it to a safe location.

2.
Once you've chosen "Backup...", the Back Up Database dialog prompts you for a number of options. While there are many options that your DBA can explain, let's just make a simple backup.

3.
Fill in the name of your database (it should already be set), and choose a backup type. In this case, we're doing a full backup. You can also perform a "differential" backup that records only the changes made since the last backup.

4.
Choose "Database" as the backup component, choose a name for the Backup set, fill in a description, and choose 0 for the expiration date (which is no expiration).

5.
For the disk location, you're going to want to click on Add to point to where you want the backup file saved. Choose a convenient location because you're also going to want to move that backup file off-site to protect it. At this point, your dialog should look like Figure 2.92.

Figure 2.92. Completing the Back Up Database dialog.

[View full size image]

6.
Click OK to proceed with the backup.

7.
Once the database is backed up to the specified file location, you'll get a promptafter which you can copy the file to a safe location.

Note that SQL Server can save backups incrementally, so instead of overlaying each backup with the previous, you can simply create delta backups and restore back to any specific point in time. Again, check with BOL or your DBA for details. Note that this backup does not back up the log, as I'll discuss in the next section.


Working with the Transaction Log

Each change made to the database is first written to the transaction logas all operations are considered to be part of an "implied," if not explicit, transaction. The transaction log is akin to an audit trail of operations made to the databaseregardless of where the changes originate. When SQL Server thinks it's appropriate, it commits changes to the database by physically writing to the hard disk and notes that the change has been "committed" in the transaction log. This change log paradigm is designed to permit SQL Server to support an extremely high volume of changes without worrying about disk latency and the power failing before the changes are persisted from the logical database in RAM to the physical database on disk.

As you can imagine, the transaction log can grow quite a bit over time, as it contains each and every change to the data or the schema for all databases except TempDB and changes made with DTS/SSIS or BulkCopy. As a result, you should plan to periodically dump the transaction log to external media and take it off-site for protectionalong with your database backups. Just how often this dump is done is a function of the volume of transactions and the volatility of your data. I've seen transaction log dumps scheduled many times a day to several times a month in production databases. Coupled with a full database backup, you can completely restore your database state, if necessary. Yes, you do need to do a full database backup when you first build your system and periodically thereafter. The transaction logs by themselves are not enough to restore your database.

In a development environment, you might find it easier to simply back up the database completely and dump (discard) the transaction log. Truncating the log can be accomplished with a single T-SQL commandit's done through the BACKUP command, as shown in Figure 2.93.

Figure 2.93. Truncating the Transaction Log.

Again, consult your DBA or BOL for more information about managing your database backups and transaction logs.


Summary

Yes, this has been a long chapter and, no, there's lots more to know about SQL Server that I simply don't have the bandwidth to pass on herenot without completely re-creating the documentation (much of which is pretty good). However, there is more than enough information here to help you get a healthy start to using and configuring SQL Server. I've touched on many of the standard features found in all versions of SQL Server, as I tried to focus on those features you're most likely to use. I talked about versions, setup, writing T-SQL, selecting data, making changes, and managing your database. Subsequent chapters leverage the understanding of SQL Server you gained from this chapter. As I said, there are a lot more features in SQL Server 2005, so I suggest that before reinventing any wheels, you should check out BOL or simply go up to the support web site and ask. I or some other reader might be able to help you directly.


Chapter 3. Relational Databases 101

    Introduction

    Getting Started with Solid Database Design

    Understanding Relational Database Normalization

    Creating Tables, Rows, and Columns

    Summary


Introduction

Many of my readers come from backgrounds that don't include formal training on the best ways to design and create efficient, business-class relational databases. If you arrive here with Microsoft Access or FoxPro experience, you're at an advantageyou know that, for the most part, the process of creating a database is hidden from you by the application's IDEyou just use drag-and-drop or use wizards to build the databases and tables you want. That's not at all bad, but without an in-depth understanding of how to best create, tune, and protect a relational database, I suspect that the relational "normalcy," relational integrity constraints, performance, and scalability of the result might not be particularly stellar. And, more importantly, the data might not be particularly secure. By "normalcy," I mean how well the database conforms to the recognized standards of relational design where database designers attempt to "normalize" databases to at least the third level. If you're not sure how to do this, or even what this means, have no fearI'll explain this later. The SQL gurus with whom I work (like Peter Blackburn, Kimberly Tripp, and a litany of others) are convinced that more problems can be solved by efficient database design than by the cleverest, best-written application front-end.

IMHO

It's not how fast you ask the questionit's how long it takes to find the answer that gates performance.


Getting Started with Solid Database Design

The Microsoft Books Online (BOL) documentation seems to fall a bit short in this important subject, so this chapter might be helpful for those who need a more complete understanding of how to create a best-practice relational database. The problem faced by any database designer is knowing what's going to be stored ahead of timebefore the first table is created. That's always been (and always will be) a problem. As I've said before, a customer rarely knows what they want until they don't get it.

To get started on the right foot, I recommend a good course in relational theory like Extended Relational Analysis. This can do a world of goodbut its depth is well beyond the scope of this book. In courses like this, you learn how to ask the right questions for each "entity" you expect to store in the database.

I also think that using a (big) whiteboard to lay out the database with your team (or customer) can help visualize the data. Getting everyone who is going to consume the data is essential. How I design a database for a single-focused project is very different than the way it's designed for projects where a small multitude of groups expect to consume the data. Admittedly, database development by committee is tough, and one should try to avoid those situations, but leaving town might not be an option.

Before we get into the academics of normalization, let's spend a few moments in quiet contemplation and focus on a few guiding principles. As you design your database, you should keep these basic tenants in mind:

  • Each table needs a unique identifier. That is, you need to choose one or more columns to permit SQL Server to find specific rows to return or update without including other irrelevant rows. In SQL Server, this typically means each table should have an "ID" column (typically cast as an Identity integer) that gives the row a unique (SQL Servergenerated) value. You should define this column using the Primary Key (PK) constraint (as discussed in Chapter 2, "How Does SQL Server Work?"). For example Au_ID is the unique identifier for the Authors table in the sample Biblio database.

  • Each table should store only one kind of information and not repeat information in multiple columns. As I discuss next, this is where normalization comes in. SQL Server is tuned to work with small, tight rows that contain relatively few columns. If you find your table has more than a dozen columns, you're treading off the boards and into the swamp. Remember, the largest row you can define is only 8K (not counting BLOB columns).

  • Microsoft feels that you should avoid columns that can be set to NULLI'm not so sure. That is, they feel that you should avoid columns where you might not have access to the data at all and cannot (logically) assign an arbitrary default. Each time you define a column as permitting NULL (making it "nullable"), SQL Server incurs extra overhead. It makes sense to move these columns to another table and cross-reference them to the table's PK as long as the database complexity does not get out of hand.

  • Each column needs to be defined both with the content in mind and with the constraints needed to keep it pure. It's not enough to type a column as integer and hope that the data entered therein is going to be pure. All columns, especially numeric columns, need to have (at least) CHECK constraints defined, if not TSQL rules. Columns should be defined to hold what's expected to be savedand no more. Needlessly bloating data type capacity simply chews up memory to no good purpose. If you have columns that contain text, but that text is never expressed in Unicode, don't use a Unicode type. If you have a date column but don't need to store time with 3.33-millisecond accuracy, use smalldatetime instead of datetime. You get the idea. In this case, less is moremore space saved, more memory available to store other pertinent stuff. I'll show you how much each column costs in memory near the end of this chapter.

    IMHO

    Understand that all data is evil until proven innocentit's not in the U.S. Constitution, but given the state of the current Congress, it just might get there.

  • Start thinking about a concurrency strategy from the beginning. Determine how data is to be shared (if at all). Consider that many "single-user" databases are doomed to failure once they are "converted" to multi-user. Think about the mechanism you're planning to use to determine if a row has changed once it's fetched. For example, you might (perhaps ought) to use a "Rowversion" or "TimeStamp" column to help track accessit can make Visual Studio's job a lot easier (and yours, too) when it comes time to write action commands to change the database.

  • Avoid BLOBs in the database. I have been suggesting this for over a decade, and those who have listened have been able to build a smaller, faster, and simpler database. If you have BLOBs, store them in files (or on RO media) and use the database to store the path.

  • Finally, and perhaps most importantly, for less-experienced developers, I think that you should strive to keep your database simplesimple to understand, support, and maintain. Excessive complexity is the bane of many a mature and amateur database designer.


Understanding Relational Database Normalization

In a nutshell, building a "good" optimized, relational database is mostly about normalization. Once you understand the basic principles of normalization, SQL Server should be able to manage your data more efficiently, the applications you write should be able to return data more efficiently, and you'll find it a lot easier to protect your database's data and relational integrity.

So, what is "normalization" and how does it help performance and all that other good stuff? Well, normalization is simply the set of relational database techniques developed to efficiently organize the information you want to manage in a relational database. The academics talk about (at least) five "normal forms," but most database designers stick to the first three forms and seldom go further. The benefit of implementing further levels is usually not that great when compared to the costsespecially in smaller databases.

Here are basic tenants of the first three normal forms.

  1. First normal: Don't define duplicate columns in the same table. Each column in a table should contain "different" information. This does not mean avoiding use of identical column names (that's prohibited by the SQL engine), but it does mean that any two columns should not store basically the same information. For example, don't create a table with two or more addresses for a customer (such as a home and business address), as shown in Figure 3.1. The solution to this problem is best implemented by the second normal form.

    Figure 3.1. Unnormalized Customers table.

    [View full size image]

  2. Second normal: All attributes (columns) in a table that are not dependent on the primary key must be eliminated. This means you need to create a separate table for each logical group of data and identify each row with a unique set of columns (its own primary key). In this case, create a separate Addresses table and connect the two tables together with their primary keys, as shown in Figure 3.2.

    Figure 3.2. Normalized Customers and Addresses tables.

  3. Third normal: Tables cannot include duplicate information. For example, if two tables require a common field, a separate table should be created to manage that column. Our basic design already conforms to the third normal form. However, as an example, take a look at the Biblio database. In this case, I created a TitleAuthor table that contains fields common to the Titles and Authors table.

No, this is not an in-depth discussion of normalization or relational theory, but it's enough to get you started. It's also important to know that many database developers bend these rules from time to time to get more efficiency out of their databases. Sometimes, they add a bit more detail for an entity in a parent table, so it's not always necessary to JOIN to another table just to get one or two bits of information. Yes, these changes mean that the data must be kept current in two different tables, and if someone else comes along and does not realize what's going on....

Understand as well that stored procedures or object-based approaches can (and do) help resolve these issues. By blocking direct access to base tables, developers can write server-side code to dereference the data in the base tables and get away with some tactics that would cause quite a bit of trouble if direct table access were permitted.

Once you have decided what tables you need, you need to use one of the SQL Server or Visual Studio tools to create them (as I illustrate in Chapter 4, "Getting Started with Visual Studio"). But before doing that, I often draw these tables on a whiteboard, which makes it easier to "see" how the data is to be stored and how the tables are related. In Visual Studio, you can use the database diagramming tool to help at this phase, and the ink does not stain your fingers as much.


Creating Tables, Rows, and Columns

Relational databases are defined in fairly simple terms[1]:

[1] See C. J. Date, An Introduction to Database Systems (Volume 1, 4th Edition) (Addison-Wesley, 1986), p. 117.

  • Database: An extensible collection of related data typically organized as a set of tables. For example, an accounting database would contain information about the customers, inventory, orders, items, and other details of the accounting operation. Where and how the data is stored, protected, fetched, and updated is irrelevant, as it is managed entirely by the database management system (DBMS). The key word here is "extensible." Due to its fundamental design, a relational database is easily expanded to encompass more data entities to store.

  • Tables: An extensible collection of rows containing related data. For example, the Customers table would contain a collection of rows pertaining to (just) customersnot the things they order, or sell, or where they bankjust about the individual customer.

  • Rows: An extensible collection of column headings and typed columns to contain data details collected in a single table. Each row pertains to a single entity as a row in the Customers table would refer to a single customer. There is never an implied row order in a relational data tablethis means, unless specifically requested, rows are returned in a nondeterministic order. This is especially true of SQL Server 2005 querieswith parallel processing, even rows stored with a clustered index can appear in any order.

  • Columns: A named storage place for base-typed, user-defined typed, or (in the case of SQL Server) sql_variant "morphing" typed data. Columns are always returned in the order in which they are defined unless otherwise requested.

IMHO

No, an ADO.NET DataTable or TableAdapter object is not synonymous with a database table.

How SQL Server Stores Relational Databases

SQL Server has expanded the number and type of objects managed and contained in the database to include collections of other objects such as logins, roles, users, stored procedures, views, triggers, functions, user-defined types, reports, and other objects; and in SQL Server 2005, assemblies, functions, aggregates, and CLR-based user-defined types (UDTs). In SQL Server, the definition of a column is expanded to include the ability to define columns whose datatype morphs to the datatype of the data stored on a row-by-row basis (sql_variant) or is defined by a CLR-based user-defined type.

SQL Server databases can contain billions of tables; tables have zero to virtually any number of rows, and rows contain 1 to 1,024 columns[2] but are (generally) limited in size to 8K[3] (not counting BLOB and variable-length columns)[4]. But, no, I don't expect your database to have more than a few dozen to a few hundred tables. If you have more than a thousand tables, you have a very complex database. I guess SQL Server supports a virtually unlimited number of tables so Microsoft could say that SQL Server supports as many tables as Oracle or one of its other competitors. It's like saying your car can contain a billion marblesjust how many marbles does one car need to carry?

[2] In SQL Server 2005.

[3] SQL Server 2005 supports row-overflow storage, which enables variable-length columns to be pushed off-row. Only a 24-byte root is stored in the main record for variable-length columns pushed out of row. Because of this, the effective row limit is higher than in previous releases of SQL Server. For more information, see the "Row-Overflow Data Exceeding 8KB" topic in SQL Server 2005 BOL.

[4] See "Maximum Capacity Specifications for SQL Server 2005" in BOL.

IMHO

If you find yourself working with a table that contains several hundred columns, there's usually something wrong with the design. Consider that the maximum size for a row (the sum of all data consumed by its columns) is about 8,000 bytes. Sure, some column data is stored on separate pages (like BLOBs and varchar(max) columns) and don't contribute (very much) to the total. Just make sure that your database is properly normalized before coming back to us when your rows are too large.

Your data is ultimately stored in named and typed "columns." The term "column" is synonymous with a "field" in an Index Sequential (ISAM) database like JET or a flat-file database. Okay, let's go over those objects in a bit more detail.

Identifiers

The database, its "owner" (the user or schema that created the object), the table, and the columns are all referenced (addressed) using SQL Server identifier object names. These names can be up to 128 bytes in length, but I generally keep the names short. I don't encourage anyone to embed spaces in the name, as it trips up the tools and your codeI also won't support you if you do. Yes, you can name your column "Customer Last Name," but you'll need to surround this column name (or any object name that contains spaces) with square brackets: "[Customer Last Name]." Most of the tools do this anyway to protect themselves from folks that insist on using embedded spaces. I'm not nearly as tolerant. I prefer to separate these long names using the underscore ("_") character or by using CamelCase, as in "CustomerLastName".

When addressing a table in SQL Server and the server is named "Fred\SS1", the database is "Biblio" and the schema[5] is "Dev1", you could address the "Sales" column in the "Customers" table by using the following identifier:

[5] I discuss the term "schema" later in this chapter.

Fred1\SS1.Biblio.Dev1.Sales.Customers

Identifiers are case-sensitive only if you install your server in case-sensitive modeI rarely do and I never encourage customers to do so. Installing an SQL Server as non-case-sensitive means you can define your columns using your company's standard naming convention and not have to worry about the case.

No, you won't be able to use special characters such as "-[]{}\|;:'"<,>.!@#$%^&*( )+=" in any identifier. You'll also discover that there is a long list of "reserved" keywords that can't (should not) be used as object identifiers. This means you can't call a database "Authorization", name a column "Sort", or name a Table "Select".[6] It also turns out that the ANSI SQL standards body has defined even more names that are not yet reserved words in SQL Server. I would stay away from these, too. Actually, if you create compound names separated by an underscore (_) character, you should be safe with virtually any name. When I get to naming stored procedures a bit later, I'll also show why using "sp_" as a prefix for a stored procedure name is a bad ideait forces the server to search for your stored procedure in the master database before looking in the current catalog.

[6] See "Reserved Keywords" in Books Online.

Defining a Primary Key

When you define your table, you need to decide how to uniquely identify each row. No, this is not an absolute requirement, but it's unusual to have a table where each row cannot be located on its own. Ninety-nine percent of the business databases I've worked with over the years define one or more columns as the "primary key" (PK) for each table in the database. In some cases, there is no formally defined PK, but one could uniquely identify a row using one or more columns.

Using a person's name as the PK might be tempting, but as your database grows, there's an excellent chance that two or more people with the name "John Smith" will show up. Even when you're building a table for individuals, you might not want to (or might not be permitted to[7]) use the (U.S.) federal Social Security Account Number (SSAN) as a unique identifier. Frankly, I think it's a mistake to do so for a number of reasons. First, this is a very important piece of personal information that could mean an individual can have their identity stolen. Second, you need to consider that the SSAN is not a unique number. While the U.S. government does not (intentionally) assign duplicate SSANs, there are other nefarious individuals ("evil-doers") "issuing" SSANs to folks needing IDs to get jobs or credit. Third, SSANs are not given to everyone in the worldat least, not yet. Using a driver's license number is also not a good idea, for the same reasons. I expect that there will be a "DNA" ID before long that will help identify peopleuntil someone shows up with a stolen thumb.

[7] The Privacy Act of 1974 states: (Sec. 7(a) (1)) "It shall be unlawful for any Federal, State, or local government agency to deny to any individual any right, benefit, or privilege provided by law because of such individual's refusal to disclose his social security account number."

Using Identity or GUID Primary Keys

Virtually all of the databases I work with use a system-generated "identity" column or a globally unique identifier (GUID) (using the uniqueidentifier datatype) as the primary key in each table. For now, let's consider use of identity or GUID columns as the best choice for your primary key. What's the difference between the two? Well, the identity column is an integer that's generated for you by the server (and guaranteed to be unique in the scope of the table), and the GUID is a unique string that you ask the system to generate in code. It's also guaranteed to be unique, but globally (all over the world). Each of these primary keys has issues when it comes to using them in ADO.NET, as I discuss in Chapter 13, "Managing SQL Server CLR Executables." Unique identifiers also have an impact on your design as well. Consider these points:

  • An identity column (integer) can be inspected, selected, or entered by your application's user far easier than a GUID. If you plan to let the user enter (I frown on this) or choose a PK from a list, you'll find that GUIDs are very hard to enter (correctly) and just as hard to pick from a list.

  • If your data is spread out over a number of servers, the GUID is the best choice. That's because you can be guaranteed that the number generated in the remote site will be different from those generated locally or from other remote sites.

  • GUIDs tend to spread out the index values more than integer identity values so data can be distributed more evenly across data pages. However, "sequential" retrieval is more expensive.

  • The GUID is a far larger value, which is somewhat more expensive to store and manage in memory.

  • You can use the NEWID TSQL function or .NET functions to set or initialize a GUID. Note that you won't get the same value twice.

  • You can also create your own GUID string, as long as it's in the format (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, in which each x is a hexadecimal digit in the range 09 or AF). For example, 6F9619FF-8B86-D011-B42D-00C04FC964FF is a valid uniqueidentifier value.

  • You can use the SCOPE_IDENTITY(), @@Identity, or other TSQL commands/functions to fetch the most recently created identity value to pass back to your client. I discuss these functions in Chapter 13 and show how to use them with an ADO.NET Update.

  • Identity column values can also be set manually, and you can set the autoincrement and seed (starting) value in code. You can also use this technique to manage client-side identity values that need to tie parent and child tables together relationally but still be able to permit the server to generate "actual" identity values when you post data to the database. I also discuss this in detail in Chapter 13.

Setting Multi-Column Primary Keys

In more sophisticated databases, as you define your table, you'll find it necessary to uniquely identify a row using more than one column. For example, suppose you're working with a Customers, Orders, Items relational hierarchy of tables. In this case, there are many customers and each customer has zero or many orders, and each order has zero or many items. For this situation, I create three tables to store the information (as shown in Figure 3.3).

Figure 3.3. Defining multiple-column primary keys.

[View full size image]

I set up CustID as the primary key (abbreviated PK) for the Customers table and set the datatype to identity. This uniquely identifies each customer with an SQL Server-generated integer value. The OrderID in the Orders table is another identity value, but I need the CustID to point to the customer that placed this order. These two columns taken together form the PK for the Orders table. Likewise, the items associated with a specific order made by a specific customer are kept in three columns in the Items table. Using this strategy, I can locate the customer associated with a particular item without having to know the OrderID.

Understanding Parents and Children

Note that the database diagram shows the relationships among the three tables. In this case, there is a primary key/foreign key (PK/FK) relationship between the Orders and Customers table, as well as the Items and Customers table. A PK/FK relationship ties two tables together, in that when a row is added to the foreign key table, there is a corresponding row in the primary key table. This means you can't add an order with an invalid or missing customer ID (CustID). Because both of these tables (most tables) have a primary key, it can be bit confusing. For this reason (and other reasons), I call the "primary key" table the "parent" and the foreign key table the "child." In our design, the Customer table is the parent, and it has two childrenthe Orders and Items tables. I could also create a tiered parent/child hierarchy, as shown in Figure 3.4.

Figure 3.4. A parent/child relationship tree.

[View full size image]

Tip

These diagrams are annotated screenshots from a database diagram created by Visual Studio.

These relationships can be defined in the database to ensure that no order is created without a valid CustID and no item is created without a valid OrderID and a valid CustID. These defined (and server-enforced) relationships are called "constraints" and are used to maintain "referential" and data integrity. When these constraints are enabled, they mean that you won't be able to delete customers from the database who have orders or items. When I start making changes in the database with ADO.NET, I'll see how I have to handle these relationships with care. Note that once these relationships are defined in the database, no matter what applications access the database, these relationships are enforced. This means you can be (more) confident that when the pointy-haired manager starts to make changes to the data with Access, he (or she) won't be able to break the referential integrityor at least, not easily.

Changing the Primary Key

One other point before I move on. Once a primary key is created, it should be considered inviolate. If you think that a change to the primary key is necessary, think again. It's far safer and easier to delete the current hierarchy and rebuild it rather than simply trying to change a primary key. If the constraints are in place (and you can disable them in code), the server won't let you change the PK until all related dependencies are removed. That means you'll need to delete all of the parent's children (and all of the grandchildren) before changing or deleting the parent row. Since the parent might have a dozens of dependencies throughout the database, this is not an easy task.

Naming Objects

There are a few things to watch out for as you name databases, tables, columns, or any other object in the database. In TSQL jargon, object names are called "identifiers," in case you want to look this up in BOL. These identifiers are created when the object is created and stored in the bowels of the master or user database. A handy place to find these names is the sysobjects tableif it still exists. The identifier specification breaks objects down into two groups: "regular" and "delimited" identifiers. The only real difference is that if the identifier does not comply with the rules for creating identifiers, it must be bracketed with double quotes or the bracket ([ ]) symbols. For example, "This is a column name" and [This is another column name] are delimited identifiers.

  • Object names must be unique in their scope. That is, database names must be unique, table names must be unique within a database, column names must be unique within a table, and so on.

    IMHO

    I suggest you code table names plural. For example, "Customers", "Orders", "Addresses".

  • The first character must be a letter from a to z or A to Z, or the underscore (_), "at" sign (@), or number sign (#). Yes, you can use Unicode[8] letter characters.

    [8] Unicode Standard 2.0.

  • Subsequent characters in an identifier can be any Unicode letter, decimal numbers, the underscore (_), "at" sign (@), or number sign (#).

  • Don't use embedded spaces in object names. Yes, you can define table and other object names that contain spaces in Access and in some of the tools (including Visual Studio), but SQL Server frowns on it. The problem is that every step along the way, you'll wish you had taken the time to remove the spaces. Each time you define a SQL query, you'll have to remember to bracket the long name so the TSQL parser and the development tools and wizards won't get confused.

  • Stay away from reserved words. This means you can't use the word "Name" to refer to a customer's namenot without taking special care[9] when you write your TSQL queries. Every time SQL Server ships, the reserved words list grows longer. Microsoft actually includes a list of words they plan to reserve in future versions, so it's not a good idea to use these, either. Look up "Reserved Keywords in TSQL" in BOL to get a complete list. It's usually (but not always) safe to concatenate two words together with an underscore, such as "Name_Like".

    [9] Look up "Quoted Identifiers" for more information. It's actually easier to avoid using these names in the first place.

  • Capitalization does not matter in TSQL identifiersunless you configure your server to be case-sensitive[10]. I often define identifiers using CamelCase notation, where each word in the identifier is capitalized. This improves readability and helps get around the reserved word restrictions. Capitalization does matter with CLR object namesbut I'll get to that later.

    [10] It's not necessary to configure your server in case-sensitive mode anymore. You can write individual queries or define specific columns to be case-sensitive.

  • Most types of identifiers have length restrictions. If you stay under 117 characters for identifiers, you'll stay on safe ground.

When naming stored procedures, don't begin the name with "sp_". Doing so tells the server that the procedure is a "system" procedure, so it takes longer to locate the object.

Tables contain one or more columns whose properties define what's to be stored therein and how the table is to be addressed when you want to return data from the table. The basic properties include:

  • The column name (identifier): I suggest choosing a name (without spaces) that clearly describes the contents of the column. The name cannot be longer than 128 characters. There are many schools of thought that dictate how tables and columns should be named, and if you work in a shop of any size at all, you'll find that your development team has already settled on a standard of some kind. It could be a "Hungarian" convention that dictates that the first few letters specify the datatype (intDaysInProduction), or some convention that your development team has adopted. Timestamp columns should be named "timestamp".

  • The column datatype: This determines how much space the column consumes in the database, and the "type" of data it's permitted to store. No, choosing the right datatype is not nearly as important as it was for typical DBMS implementations, as hard disks are far larger than ever; and for small databases, space considerations should be well down on the list of concerns. However, for larger databases or those with high-volume demands, reducing the size of column footprint can help performance. In any case, I usually define the datatype to handle strings, numbers, graphics, or XML. I'll discuss the rudiments of choosing the right datatype in the next few pages.

  • Specifying a user-defined datatype: When I create tables, I sometimes do so by specifying custom, user-defined datatypes. This way, I can define rules and defaults on these columns that apply to the entire database. See the discussion on UDTs, rules, and defaults later in this chapter. They are also discussed in Chapter 2.

  • If the column is permitted to accept NULL values: If you expect to store information that might not be known as the row is added (like "Date_Married"), you need to define the column as permitting NULLs. However, you can't define a column that's going to be the table's primary key as permitting a NULL value.

  • Is the column the primary key? If this column is the primary key (or one of the columns that together constitute the primary key), you can so indicate. You can also indicate if the PK is to be kept unique within the table.

  • Is the column value to be generated as an "Identity" value? As described earlier in this chapter, SQL Server can automatically generate a primary key value for your tablejust request that the column be designated as IDENTITY.

  • Is the column value to contain a GUID? In this case, request that the server designate the column as ROWGUIDCOL (using the uniqueidentifier datatype)you might want to generate the GUID yourself as new rows are added, but it's easier to use the NEWID function as the default column value.

  • How should the column be collated? You can specify the dictionary order, case-sensitivity, and accent-sensitivity (especially if it is different from the collation specified for the database). This means you can define columns holding a name as case-sensitive and others as non-case-sensitive (or leave them to default to the database collation sequence). The collation also determines how the data is sorted. This is important for anyone working with Unicode data or character sets that don't sort the same way as the database default. See "Using SQL Collations" in BOL for more information.

  • What action should be taken when a row is deleted or updated? By setting the ON DELETE and ON UPDATE attributes, you can get SQL Server to implement cascading deletes or updates.

Sure, there are many other options you can specify as you define your table, but the options shown here are enough to get you started. This process needs to be repeated for each column in the table and for each table in the database.

Frankly, I expect that most of you will use the Visual Studio or Management Studio tools to define tables. You'll find it's pretty easy to define your tables, primary keys, and relationships using the interactive Database Diagram tool in Visual Studio. Your other alternative is to figure out which TSQL or SMO commands are required to configure a new table (or alter an existing table). If you're getting paid by the hour, this is your best bet. All kidding aside, some folks really like the approach of creating scripts to record how their tables are defined. Fortunately, the tools can do that, toothey can take an existing database and write a file that includes all of the TSQL needed to build it up from scratch. Sure, you're going to have to add data on your own.

Using User-Defined Types, Rules, and Defaults

In SQL Server, non-CLR UDTs are pretty straightforwardthey're simply aliases to the base types[11]. This way, you can define a UDT for "PostalCode" (based on a varchar(11) and specify the PostalCode UDT when the table is created. Once defined, a UDT can be assigned a global default. That is, when a new row is added to the table and no value is supplied, SQL Server substitutes the registered default for the column value and any other columns defined with the UDT.

[11] CLR user-defined types are far more complex. I'll defer the discussion of those to Chapter 13.

In a similar manner, you can also define SQL Server rules[12] or (better yet) check constraints for specific columns or to UDTs, as I discussed in Chapter 2. These constraints are used to implement your business rulesthey define what's permissible in a specific column and what's not. For example, you know (based on how you run your business) that customer discounts can range from 0% to 15% and correct shipping delays are between 1 and 90 days. Setting up SQL Server rules to enforce these business rules is fairly simplecheck constraints are a bit harder. Both rules and constraints can be any expression valid in a WHERE clause and can include such elements as arithmetic operators, relational operators, and predicates (for example, IN, LIKE, BETWEEN). However, the constraints cannot reference columns or other database objects. Let's walk through the process of creating a new UDT (alias) and associated check constraints.

[12] According to BOL, CREATE RULE will be removed in a future version of SQL Server. Microsoft suggests that I avoid using CREATE RULE in new development work and plan to modify applications that currently use it. I don't think Microsoft can drop rules anytime soon without causing a riot, so I wouldn't worry too much just yet.

Start by creating a new User-Defined data type in the database by using the SQL Server Management Studio wizard that starts when you right-click on User-defined Data Types | New User-defined Data Type, as shown in Figure 3.5.

Figure 3.5. Creating a new User-Defined data type.

All I have to do is fill in the form, as shown in Figure 3.6. Here, you provide the UDT name, base datatype, and length. You can also specify that the UDT can be set to NULL. Later, I'll use this same dialog to set the default value and the rule/check constraint for this type.

Figure 3.6. Creating a new UDT based on the varchar datatype.

Next, I create a new constraint for our PostalCode UDT, as shown in Figure 3.7. Again, right-click the Constraints item under the selected table.

Figure 3.7. Adding a New Constraint for the PostalCode UDT.

Creating Table Indexes

Once you define your database tables and the primary key, you're going to want to add indexes to improve query performance. Without indexes, you'll find that query performance is rather slow. If you take a look at the query plan being generated, you might find that SQL Server is not fetching rows efficiently by scanning the entire table each time. Again, the interactive Database Designer tool can help set up indexes. No, don't add too many, as each index must be updated as you insert new rows. Start with an index on the primary key columnsthe tools should do that for you automatically. Once you populate your database with data, you can run the query analyzer to evaluate your indexes. This tool will tell you which indexes are helping and which are not, as well as where additional indexes will further improve performance.

Choosing the Right Data Type

When designing databases in the 1960s and 1970s, I was taught to be especially careful of how much space each data element consumed. Since hard disks were tiny by today's standards (the IBM 360 125 came with 7.25Mb to 100Mb drives),[13] I was hard-pressed to minimize the amount of data stored in each "record." I economized by "coding" whenever and wherever I could. For example, a single column (byte) might contain several different types of data, depending on the value to be stored. When SQL Server and other relational databases were introduced, disk space was still expensive, but not nearly as much as in the mainframe days. However, more experienced database architects still choose column widths based on past experience and with the knowledge that more data means poorer performance.

[13] See www-03.ibm.com/ibm/history/exhibits/mainframe/mainframe_PP3125.html

Unicode vs. ANSI

In situations where you need to store data in an "international" character set, whose characters are not supported by the ANSI set, you'll have to define your columns (and string literals) as Unicode. If you take this option, SQL Server stores 16 bits for each character instead of 8. It means the same four-character entry requires 4 bytes in ANSI and 16bytes in Unicode columns. Just remember to prefix your string literals with "N", as in N'Fred', when building Unicode expressionsVisual Studio tools and wizards do this for you if they generate the query. There is another aspect to Unicode that might surprise you. When you define a column as nvarchar, you specify a maximum length, as shown in Figure 3.8.

Figure 3.8. Creating a table with a Unicode column.

This DDL code allocates 50 bytes of space in the data row to the Author's name. However, this also means that the name must be no longer than 25 (16-bit) characters.

Char vs. VarChar

I've heard the debates over use of fixed-length datatypes (like char and nchar) over variable-length types (like varchar and nvarchar). In my practices, I rarely use the fixed-width types because they're problematic in a number of respects. These types are fine for columns whose data is always the same number of characters, but if you slip and provide a value that's shorter (or longer) than the defined size, SQL Server either pads the remaining space or truncates the data (often without notice). You'll also find that it's tough to create expressions against fixed-length columns unless you match the length of both operands. For example, if your fixed-length column can contain four characters, you'll have to write an expression that has exactly four characters, or an equality expression will always return False.

IF MyFixedCol = 'Fred' | This returns TRUE if MyFixedCol contains "Fred".
IF MyFixedCol = 'Fred' | But this returns FALSE.

For this reason (and others), I prefer to use variable-length types. They don't consume much extra space (if any) and when the data length varies, this approach can actually save space. When defining variable-length character columns, you specify the maximum amount of space to reserve for the column. This does not preallocate this spaceit simply sets an upper limit. With SQL Server 2005, you can now define a varchar(max) or nvarchar(max) column that (like the TEXT datatype) can store up to 231 bytes and Unicode 230 bytes.

Decimal vs. Floating Point

When you record a money value in the database, it's best to understand the nature of the values you intend to storeespecially the precision. For those of you that took computer science in school, you know that it's not possible to store some values in binary. For example, [1/3] is stored as .3333 (with a never-ending list of "3"s.) While the value might be close, if you add [1/3] + [1/3] + [1/3], you'll get .9999you've lost some precision. Sure, with rounding, the result is returned as 1, but in some cases, you aren't permitted to round.

IMHO

In the early days of computing, clever programmers were able to strip off the extra precision (values less than a penny) and salt it away in another account. By the end of the week, they had accumulated a tidy sumespecially when millions of dollars were changing hands.

When you store a money value, be sure everyone knows the currency on which this value is based. This can help you from making a mistake when bidding on a project in the U.K., where the dollar is worth (at today's rate) about £0.529269. You might consider using a CLR-based user-defined type to keep the currency type stored with the valueespecially if a single column can hold values from more than one currency.

The decimal datatype is listed (as shown in Table 1.1) under "Exact Numerics". That is, it's designed to hold an exact value. When you declare a decimal or numeric (they are equivalent), you also can declare the precision and scale (it defaults to 18). The precision is the maximum total number of decimal digits that can be storedincluding the values on either side of the decimal point. To store a value of 1234.1234, you would need a precision of 8.

The scale indicates the maximum number of decimal digits that can be stored to the right of the decimal pointthis must be a value from 0 to the defined precision. The default scale is 0, so unless you define a scale, your value will be stored as a whole number (without a decimal portion). You won't be able to define a scale unless you define a precision as well. For example (as shown in Figure 3.9), to define a column with a precision of 10 and four decimal places, you would code:

Figure 3.9. Declaring a decimal column with specific precision and scale.

Working with Imprecise Numbers

When working with scientific data where you need more precision but not 100% accuracy (which sounds a bit strange), you can choose the approximate number data types. Sure, some numbers can be expressed exactly, but others can't due to binary round-off. In the case of the float datatype, you can define the precision and storage size by providing a value that determines the number of bits used to store the mantissa[14] of the floating point number (in scientific notation). If you supply a value between 1 and 24, the float's precision is set to 7, and it takes 4 bytes to store the value. If you provide a value between 25 and 53, the float's precision is set to 15, and it takes 8 bytes to store the value. The default is 53. Note that SQL Server 2005 resets the mantissa setting to either 1 or 53, based on the value you supply.

[14] Mantissa: the fractional part of a floating-point number.

Table 3.1. SQL Server Datatypes and Their Precision

Datatype

Bytes

Exact Numerics

These values are stored so the value stored is expressed exactlythey are not subject to binary round-off.

Integers

bigint

8

Integer (whole number) data from 2^63 (9223372036854775808) through 2^631 (9223372036854775807).

int

4

Integer r(whole number) data from 2^31 (2,147,483,648) through 2^31 1 (2,147,483,647).

smallint

2

Integer data from 2^15 (32,768) through 2^15 1 (32,767).

tinyint

1

Integer data from 0 through 255.

Bit

bit

1

Integer data with either a 1 (True), 0 (False), or NULL value.

Decimal

decimal

517

Fixed precision and scale numeric data from 10^38 +1 through 10^38 1.

numeric

Functionally equivalent to decimal.

Money

money

4

Monetary data values from 2^63 (922,337,203,685,477.5808) through 2^63 1 (+922,337,203,685,477.5807), with accuracy to a ten-thousandth of a monetary unit.

smallmoney

8

Monetary data values from 214,748.3648 through +214,748.3647, with accuracy to a ten-thousandth of a monetary unit.

Approximate Numerics

These values are stored in binary and are used when a precise but not 100% accurate value must be stored.

float

48

Floating precision number data from 1.79E + 308 through 1.79E + 308.

doubleprecision

8

Equivalent to float(53) (8 bytes).

real

4

Floating precision number data from 3.40E + 38 through 3.40E + 38.

Dates

datetime

8

Date and time data from January 1, 1753, through December 31, 9999, with an accuracy of three-hundredths of a second, or 3.33 milliseconds.

smalldatetime

4

Date and time data from January 1, 1900, through June 6, 2079, with an accuracy of 1 minute.

ANSI Character Strings

These values are stored as strings of characters in non-Unicode (ANSI) encoding (8-bits/character).

char

N

Fixed-length non-Unicode character data with a maximum length of 8,000 characters.

varchar

N

Variable-length non-Unicode data with a maximum of 8,000 characters.

varchar(max)

N

Variable-length non-Unicode data with a maximum length of 2^31 1 (2,147,483,647) characters.

text

N

Variable-length non-Unicode data with a maximum length of 2^31 1 (2,147,483,647) characters.

Unicode Character Strings

These values are stored in Unicode (16-bits/character).

nchar

N

Fixed-length Unicode data with a maximum length of 4,000 characters; 16 bits stored for each character.

nvarchar

N

Variable-length Unicode data with a maximum length of 4,000 characters.

sysname

128

System-supplied user-defined data type that is functionally equivalent to nvarchar (128) and is used to reference database object names.

nvarchar(max)

Variable-length Unicode data with a maximum length of 2^30 1 (1,073,741,823) characters.

ntext

N

Variable-length Unicode data with a maximum length of 2^30 1 (1,073,741,823) characters.

Binary Strings

These values are stored in binary with any attempt to encode them.

binary

N

Fixed-length binary data with a maximum length of 8,000 bytes.

varbinary

N

Variable-length binary data with a maximum length of 8,000 bytes.

varbinary(max)

Variable-length binary data with a maximum length of 2^31 1 (2,147,483,647) bytes.

image

N

Variable-length binary data with a maximum length of 2^31 1 (2,147,483,647) bytes.

Other Types

cursor

A reference to a server-side CURSOR.

sql_variant

N

A data type that stores values of various SQL Server-supported data types, except text, ntext, timestamp, and sql_variant.

table

A special data type used to store a rowset for later processing.

timestamp

8

A database-wide unique number that gets updated every time a row gets updated.

uniqueidentifier

16

A globally unique identifier (GUID).

xml

N

Names an XML schema collection. Can store up to 2GB of data.

Using the xml Datatype

For the first time, SQL Server 2005 introduces the new xml datatype. This means you're going to be able to store XML data in your table's column(s). Because xml is a "real" built-in type, you'll be able to use it when creating a table as a variable type, a parameter type, or a function return type. You'll also be able to use it in CAST or CONVERT. That said, I need to discuss where it makes sense to use xml typed data columns or xml typed arguments. One interesting use would permit you to pass lists of values to be used in an IN expression. Yes, you would need to write a function to convert this to a table-type variable.

Using the sql_variant Datatype

One of SQL Server 2000's innovations was "lifted" from Visual Basicthe "variant." The sql_variant datatype is unusual, in that it's designed to "morph" itself to most (non-BLOB) types. This means when you define a column as sql_variant, it can contain an integer (of any size), a string, a float, money, or even an xml structure. The sql_variant column value does not take on a type until you assign a value to it. I suggest you check out BOL for the rules and regulations involving this unique type.


Summary

This chapter gives you a kick-start on designing relational databases that can perform better, be easier to maintain, and be more successful. That's usually an unspoken goal of any database application project. Unless it's successful, you'll either be back working on it when you should be home relaxing or be out looking for another jobwithout a good recommendation from your last employer. I talked about both formal rules and informal suggestions to normalize your database. Understand that few of these rules are cast in stone, but until you fully understand them, the databases you (and your team) design and implement, populate and test, and polish and deploy won't keep bread on the table.


Chapter 4. Getting Started with Visual Studio

    Introduction

    How I Got Here

    Installing the "Right" Version of Visual Studio

    Launching Visual Studio

    Customizing Visual Studio 2005

    Configuring Server (or Database) Explorers

    Creating and Managing Database Connections

    Managing Database Objects with the Server Explorer

    Managing Queries with the Query Designer

    Using the Server Explorer to View Synonyms

    Using the Server Explorer to Manage Types

    Using the Server Explorer to Manage Assemblies

    Using the Server Explorer to Manage Servers

    Summary


Introduction

This chapter introduces Visual Studio 2005 to those unfamiliar with the new version of the integrated development environment (IDE). Since there are so many new features in the 2005 edition, I expect that includes everyone. I'll start by showing you how to set up the Server Explorer and Toolbar to make it easier to get started with new applications. This chapter also walks you through the (very intuitive) Visual Studio 2005 setup program and (not so intuitive) customization routines. If you've already installed Visual Studio, you can just skim this first section. However, Visual Basic developers need to take a close look at the configuration section, as I discuss re-enabling a number of features you'll miss (and I discuss later in the book). I'll also provide a brief tour of the Visual Studio IDE so that you'll know where the toolbox, Server Explorer, and other windows are located; how to customize their appearance; and how to use these tools as you develop, test, debug, and deploy applications. I'll show you how to set up database connections to your SQL Server or other data sources, including local instances of SQL Server Express edition. Remember that this is a "data-centric" book, so I'll focus quite a bit on the "data" features of the IDE. I leave many of the other UI[1] design tools and elements to other chapters (and other books).

[1] UI: User Interface.


How I Got Here

I've been working with Microsoft "visual" tools since the first version of Visual Basic was released in 1991, over 14 years ago. Since then, Visual Basic was joined by other visual language development toolseach with its own unique characteristics and following. In our opinion, the folks at Microsoft decided it was expensive to keep developing and marketing language-specific IDEs and decided (somewhat unwisely, IMHO) to create "Visual Studio" to act as a bundled launching pad for Visual Basic and the other visual languages. At first, each language had its own language-specific developer interface, but when Visual Studio .NET 1.0 appeared in 2000, these were morphed together (more or less) into a common IDE, despite substantive differences in the development paradigms. As I see it, compromises were made at every turn to implement this approach. Since then, Visual Studio has further "evolved" and genericized so developers could tune and configure the IDE to match their specific needs and (more importantly) the familiar keyboard and window layouts to which they had grown accustomed over the years. The Visual Studio 2005 version is far more tolerant of other configurationsit supports configurable profiles to alter the keyboard shortcuts and IDE layouts. While Visual Studio 2005 has made a number of somewhat rude assumptions about Visual Basic developers, Visual Studio 2005 can be easily reconfiguredassuming you know what to ask for. I'll show you how to configure these settings in this chapter. Sadly, if you choose the "Visual Basic" profile, Microsoft assumes that you are a "hobbyist," so it hides a number of important featureslike your mommy putting away the sharp scissors and chainsaw, in case you might be tempted to saw up your sister. I'll tell you how to re-enable these features in this chapterbut please don't dismember anyone in your (or my) family.

At one point in time, I thought Microsoft had started to re-evaluate the wisdom of this OSFA[2] approach to the application-development tools. It does not take a rocket scientist to realize that a Web Service and an ASP.NET application have very little in common. If you add to that Windows Forms (what they now call "Smart Client") applications, you easily see that a common IDE doesn't really make a lot of senseexcept that it's easier to package and sell. Given that new web-centric tools have been created to compete with Visual Studio, you can understand Microsoft's motivation to break the studio apart. I doubt if this will happen anytime soonespecially given the fact that they have recently combined the ASP and Windows Forms teams into one group. The "Orcas" version (due to be released near the Vista timeframe) is pretty much designed and well along the way toward a late 2006 or early 2007 launch. Perhaps the following release ("Hawaii") will incorporate this "separate but equal" approachbut I doubt it.

[2] OSFA: One size fits all.


Installing the "Right" Version of Visual Studio

When Visual Studio first went to market, it appeared in a dizzying array of versions. These include "Express", "Standard", "Professional", and "Team" SKUs[3]. I won't attempt to describe the differences between these versions, as Microsoft has done a pretty good job of doing so[4]. Consider that I'm developing the materials for the book using the Team Edition (for Software Developers). This version exposes the (copious) project management as well as code analysis tools used to build and test the code. Nope, this edition is probably too expensive for many development shops. At almost $3,000, it's fairly pricy. While the Express editions are freely downloadable, they lack many of the features you'll need in order to create, test, and deploy serious applications. I expect you'll want to use the Standard ($200) or (more likely) the Professional Edition ($2,500). Given the prices, you might find it more cost-effective to sign up for an MSDN subscription. Many of the developers I meet at conferences are subscribersgenerally, they find the ability to download whatever tools they need on demand is a substantial benefit and worth the price. If you're a student, check with your school or instructor or that guy in the dorm that seems to have everything. Most institutions offer academic discountsexcept those that advocate use of open source. Just don't get caught with a pirated version. There's no telling what it can do to your system.

[3] SKU: Stock-keeping units. This is the term used to refer to individual products like Visual Studio 2005, Standard Edition.

[4] http://msdn.microsoft.com/vstudio/products/

Each version of Visual Studio is designed for a specific version of the .NET Framework. This means that you won't be able to reference any other version of the Framework. The original Visual Studio .NET (1.0) supports development against the 1.0 version of the .NET Framework. Visual Studio .NET 2003 supports the 1.1 Framework and Visual Studio 2005 supports the 2.0 Framework. No, you won't be able to create applications targeted to a down-level (or up-level) version of the .NET Framework. The Orcas version promises to address this (somehow), but it's not at all clear what that means in reality.

Dealing with CTP or Beta Shards

Since this book will appear long after the beta and community tech preview (CTP) releases of Visual Studio 2005, you should not have to worry about installing Visual Studio on a system that has had one of the pre-release versions installed. However, there are CTP versions of Orcas being distributed as I write this book. To be perfectly clear, you should never try to install a pre-released copy of any software (Microsoft or not) on a system that you can't afford to reformat or re-image with ease. Please understand that it's virtually impossible to uninstall software as complex as Visual Studio or SQL Serverit's like trying to remove all of the rice from your suitcases after your honeymoon. I've had mixed success using the "uninstaller" utilities that Microsoft (and others) provide. Yes, some folks report that these utilities work fine and, in one case, one of these utilities was able to clean up one of my systems (after several hours) to permit a full install. However, my concern is that if they "seem" to work but don't clear out every jagged shard of the previous installation left behind after the uninstall, your application development experience might be plagued with problems that are hard (to impossible) to nail down. Again, I recommend that you set up your Visual Studio 2005 on a "clean" systemone that has never had any beta software installed.

Walking Through a Typical Installation

The installation dialogs in Visual Studio are easier than ever to run. The Setup program (launched by autostart on the DVD) begins with a "navigator" dialog, as shown in Figure 4.1

Figure 4.1. The initial setup dialog for Visual Studio 2005.

I'm installing the Professional Edition in this walkthrough. The Express, Standard, and Team editions use different (albeit similar) setup programs. The Express setup has only two optionsinstall SQL Server Express Edition or not.

It's from here that you can install (step by step) Visual Studio and the documentation and check for updates. As Setup completes each phase, it returns to this dialog. Click on "Install Visual Studio 2005" to proceed.

As Setup continues, it checks to see if any recognizable shards of the beta or CTP versions remain on your systemno, it does not always find them all. It also checks to see if you're running one of the acceptable OS and hardware configurations.

If you're running CP/M and have figured out how to start setup, you'll find that it won't get very far.

If your system passes muster, you'll see the first dialog that asks if you want to send information about your setup experience to Microsoft (see Figure 4.2). If you choose this option, Microsoft sends a report back to the mother ship if Setup gets that far. This data (supposedly) helps Microsoft refine the setup routineseither that or it helps identify the types of systems people are trying to use or the brand of cigars you smoke.

Figure 4.2. Do you want to send information back to Microsoft?

[View full size image]

The next dialog you'll (eventually) see (Figure 4.3) reports on the setup components already installed on your system. In this case, I've already installed the "Professional" version of Visual Studio 2005, so Setup found many of the fundamental components already in placelike the 2.0 Framework. Enter the product key as provided with Visual Studio 2005 and click Nextafter you accept the terms of the End User License Agreement (EULA). Of course, I recommend that you read this contract carefully or get your attorney to do so. It has been rumored that earlier versions of the EULA contained references to your first-born child and periods of unpaid servitude.

Figure 4.3. Enter the product key provided in your Visual Studio packaging.

[View full size image]

Once you've accepted the EULA, you'll be asked to choose what type of installation you want (see Figure 4.4). I recommend that you choose "Custom". This way, you can enable or disable specific languages that you intend to use. This approach can mean a faster install and fewer irrelevant bits being dumped on your hard drive. This dialog also reports on available disk space and the amount of space your choice (might) require. You'll get another chance to set this path in the next dialog (if you choose "Custom").

Figure 4.4. Choose "Custom" to permit selection of the languages and features to install.

[View full size image]

On the "Custom" options page, you can disable installation of any of the languages or features that you don't need. In this case, I disabled the "Visual C++" language. Note that as you uncheck languages or features, the amount of disk space required for the installation is adjusted accordingly. Note that Visual Source Safe is not listed here as an option. You'll have to install that separately if your SKU includes it. If you see a green box in one of the option checkboxes, it means that only part of that feature is to be installed. You'll need to drill into each of these items to enable/disable specific components.

SQL Server Express edition is one of the options you'll see in some Visual Studio SKUs. If you choose to install it, the Setup program will take all of the default options and install a named instance (SQLEXPRESS) of SQL Server on your system. I'll discuss how to tune up that instance later in this chapter. However, if you want to customize the SQL Server Express installation (and many of you will want to), uncheck the SQL Server Express option. Once Visual Studio has installed, you can go back and run the SQL Server Express setup independentlyyou'll be able to customize the location and other attributes of the installation at that time.

Figure 4.5. Check or uncheck the features and languages you need or don't need.

[View full size image]

Once you click "Install", setup proceeds to install the selected languages and features on your hard disk. When it's finished installing, you'll get a report on any issues that remain (as shown in Figure 4.6). In this case, I choose the Office 2003 development features, but I don't have Office 2003 installed on the machine. I can ignore this warning.

Figure 4.6. The install phase of the Setup program is completed.

[View full size image]

This completes phase 1 of the installation routine. When you click "Finish", the Setup program returns to the initial navigator dialog, as shown in Figure 4.1.

Installing the MSDN Documentation

In phase 2 of the installation, you install the MSDN product documentation. Again, these dialogs let you install all or part of the documentation on your local hard drive. Note that MSDN documentation (more current documentation) is available online directly from the MSDN web sites, so the locally installed documentation might be of less interest unless you don't have live online connectivity. The dialog shown in Figure 4.7 permits you to choose from the list of MSDN documentation modules to install. Remember that you can return to this dialog later to add or remove any of these items.

Figure 4.7. Choose the documentation files to install.

[View full size image]

Once the documentation is installed (you can usually go to lunch while it's copying files), you'll return again to the navigator dialog (see Figure 4.1). At this point, you can check for service releases. Basically, this returns you to the online Windows Update site that lets your system scan for services updates that apply to your system.


Launching Visual Studio

As with any Windows application, the Visual Studio 2005 Setup program associates application solution files (.sln) and several other extensions (like .xml, .vb, .cs, and others) with the DevEnv.Exe Visual Studio executable. Except for the Express Edition, Visual Studio 2005 can be found in Start | Programs | Microsoft Visual Studio 2005 | Microsoft Visual Studio 2005. The Express edition is installed in the root Start | Programs directory.

When Visual Studio Standard Edition or better starts for the first time, if a prior installation of a version of Visual Studio (.NET or 2005) is not found, it displays a dialog that prompts you to choose the type of "developer activity". While strangely worded, basically they're asking if you want Visual Studio to behave and be configured for one of five types of "activity" as shown in Figure 4.8.

Figure 4.8. First-time execution of Visual Studio 2005 queries for "development activity".

Once you make a selection, "activity"-specific keyboard layout schemas are installed and features are made visible. No, this selection is not cast in stone, as you can load pre-configured configurations or customize a configuration and save it for later reuseand you're probably going to want to do just that.

Installing the Visual Studio 2005 Image Library

For reasons that escape us, Visual Studio setup does not install any icon files that developers can use in their applications. These are included on Standard or better edition DVDs in compressed form and copied to your system, but not placed anywhere you easily find themthey're hidden at C:\Program Files\Microsoft Visual Studio 8\Common7\VS2005ImageLibrary. This path contains a compressed file VS2005ImageLibrary.zip that should be extracted to a convenient place on your development system.


Customizing Visual Studio 2005

When Visual Studio 2005 is first installed, you'll want to further tune the configuration to meet your specific preferences. Depending on the version of Visual Studio you're accustomed to, you might want to move the individual explorer windows or make other changes that make your job easier. Let's start by using the Tools | Options menu to further refine our custom configuration.

1.
Click on the Tools | Options menu to expose the dialog shown in Figure 4.9.

Figure 4.9. The Visual Basic Tools | Options dialog.

[View full size image]

2.
Note that not nearly all of the options are exposed in this listthat's because I chose "Visual Basic Developer" as the Developer Activity. To see the remaining options, click the "Show all settings" checkbox.

3.
In Standard and better editions, scroll down in the Options list and click on "Projects and Solutions (General)." I recommend that Visual Basic developers check "Show advanced build options". This permits you to modify your debug toolbar to include the build type (debug or production), as shown in Figure 4.10.

Figure 4.10. Enable "Show advanced build configurations".

[View full size image]

4.
Next, click on the "VB Defaults" item in the Options list. Set "Option Strict:" to "On", as shown in Figure 4.11. This option ensures your Visual Basic applications must be hard-coded to force narrowing type conversions instead of letting Visual Basic perform these conversions behind the scenes. Once set, late binding is not allowed. This means you'll have to make sure that the datatype of the variable you're moving matches the destination type.

Figure 4.11. Set the Option Strict option to "On".

[View full size image]

5.
In Standard and better editions, select the "Keyboard" option and review the settings already applied. Remember, I chose "Visual Basic" application activity, which sets the keyboard shortcuts to match those used by Visual Basic 6.0well, sort of. Visual Basic 6.0 developers expect that F8 is used to "step into" an application. The default "Visual Basic 6.0" settings have this set to F11. If you want the Visual Basic debug keyboard shortcuts to work using F8, you'll need to change the Debug.StepInto shortcut, as shown in Figure 4.12.

Figure 4.12. Reset the Debug.StepInto keyboard shortcut to F8.

[View full size image]

6.
Select the "Text Editor" and drill into "All Languages". In this dialog, I enable line numbers for the source code shown in the code editor, as shown in Figure 4.13.

Figure 4.13. Enable line numbers for the source code editor.

[View full size image]

No, you don't necessarily need to make all of these changes in your configuration, but if you do, your code will compile and appear in the editor in the same way as our examples. Now that I've made these changes, it's time to save them.

Saving and Loading Custom Configurations

Whenever you apply custom settings, it's a good idea to save these configuration changes so they can be restored on demand. These custom configuration settings files can also be shared and loaded by other developers on your team so that everyone is using the same (custom) configuration. To save your settings:

1.
Click the Tools | Import and Export Settings... menu. This opens the dialog shown in Figure 4.14.

Figure 4.14. Saving your custom configuration with the Import and Export Settings Wizard.

[View full size image]

2.
Click "Next >" and select the options to export in the subsequent dialog (shown in Figure 4.15). You can select or de-select as many of the settings as needed.

Figure 4.15. Choose the settings to persist to your custom configuration file.

[View full size image]

3.
Click "Next >" to open the final file designation dialog. Name your settings file and provide a path where the settings are to be saved (as shown in Figure 4.16).

Figure 4.16. Name and provide a path to your custom file.

[View full size image]

4.
Click "Finish" to save the custom file.

As you work with Visual Studio 2005, you'll often revisit these option settings. I encourage you to save your custom configurations, so when you need to revert to a "safe" configuration, you can easily do so. If you upgrade your Visual Studio edition, you can use these same configuration files to reset your favorite settings.

Creating Custom Project Templates

In Standard and better editions, before you start creating projects, it makes sense to work up a custom template. In the Express edition, you'll have to make do with the built-in templates. Custom templates let you start a new project from your own template that's already filled in with the comments, "To-dos", references, and settings your company's applications use.

Let's walk through the process of creating a custom template and exporting it so Visual Studio can list it as one of the available templates when you start a new project.

1.
Click on File | New Project. This launches the New Project dialog, shown in Figure 4.17.

Figure 4.17. Choose a starting template.

[View full size image]

2.
Depending on the type of application template you want to build, open the appropriate Windows or other project type. I'm going to show how to create a custom Windows project template, so I chose "Windows Application". Name the project to the default name your custom projects will use. I'll use "HHGTemplate".

3.
Click "OK". This creates a new project using the settings and configuration built into the standard template.

4.
Customize the project to incorporate any standard configuration options. For example:

  • I added copyright comments to the Form1.vb file.

  • I added a Private Sub New and code to the Form1_Load to configure a SqlConnection object.

  • I also added a BuildAndExecuteCommand sub to perform a simple query. This code is used as a base for many of our examples.

    This code is shown in Figure 4.18.

    Figure 4.18. Custom code added to the custom HHGTemplate Form1.vb.

5.
No, I'm not done yet. Note that the SqlConnection class constructor references the Application settings (stored in the app.config file). The (Visual Basicspecific) My.Settings.BiblioConnectionString references a specific application setting. I need to ensure that this setting is applied to the app.config file. Yes, if you're creating an ASP.NET template, you'll want to change the Web.Config file. Right-click on the project (HHGTemplate) and choose Properties. This opens the Settings.settings file dialog, as shown in Figure 4.19.

Figure 4.19. Configuring the application settings.

[View full size image]

In this case, I need to add a custom setting that's used as a ConnectionString. Start by naming the setting and choosing a appropriate type from the drop-down list of types. In this case, I chose "(Connection string)".

6.
Next, set the Scope to "Application" and set the Value to your database connection string. This can be typed by hand, or, if the Value cell is selected, the dialog exposes an ellipsis (...) that can be selected to open a Connection Properties configuration dialog (as shown in Figure 4.20).

Figure 4.20. Opening the Connection Properties dialog.

[View full size image]

7.
The Connection Properties dialog collects all of the needed elements of the ConnectionString. In this case, I'm setting up a ConnectionString to address the DemoServer (an alias[5] for a specific server on our local network), use Windows (SSPI) authentication, and set the initial catalog to the "Biblio" database. To make sure the ConnectionString is valid, click the "Test Connection" button. Nope, it might not work unless you've done some configuration first. Remember, SQL Server is initially installed to be invisible from the network. While you might be able to easily connect to a local SQL Server instance, getting connected to an instance on the local area network (regardless of your rights) won't be easy. I discuss how it's done in Chapter 9, "Getting Started".

[5] See Chapter 2, where I show how to set up an alias to a specific SQL Server instance.

8.
Click "OK" when you have configured your Connection properties. This copies this preconfigured ConnectionString into the Settings dialog. Now is the time to add settings to your Settings.Settings (app.config or web.config) file. Note that these values are compiled and also stored in the Settings.Designer.vb file as strongly typed properties. Figure 4.22 shows how the "BiblioConnectionString" setting is compiled into a strongly typed property.

Figure 4.21. Configuring a ConnectionString.

Figure 4.22. The strongly typed BiblioConnectionString "setting" property.

[View full size image]

9.
It would be a good idea to visit each of the project property tabs while you have the dialog open. Interesting settings include:

In the Application tab:

  • Setting the application icon.

  • Automatically add code to ensure that only one instance of the application is started. (Make single instance application.)

  • Determine when application is shut down.

  • Enable a splash screen.

  • Add code to specific "startup" events

In the "Compile" tab:

  • Set "Advanced" compiler settings. Here, you can enable or disable integer overflow checks and optimizations. Note that compiler optimizations are disabled when you create a "debug" build and enabled when you create a "production" build.

  • Set Option Strict: On. I've already discussed the importance of this setting.

In the "Signing" tab:

  • When you create a new template, I recommend that you set up a signature file. Choose "<new>" from the "Choose a string name key file:" drop-down and create a new key file.

This is also a good time to add class files or modules and any custom controls your application template needs to include.

10.
You're ready to export your template, so click the Save All button or choose File | Save All.

11.
Next, click File | Export Template. I'm creating a "Project" template, so all of the files in the template are included in the new Template project. Note that when you start a new project using this template, a copy of each of these files (in their current state) is loaded into your new project. Figure 4.23 shows where to enter your template name.

Figure 4.23. Enter the name of your template in the Export Template Wizard.

[View full size image]

12.
Click Next and fill in the Template icon, name, and description, as shown in Figure 4.24. Click Finish to install the template into Visual Studio 2005's list of valid templates.

Figure 4.24. Describing the project template

[View full size image]

13.
Test your new template by creating a new project (File | New Project). Note that your custom template is listed under the languagein this case, "Visual Basic", as shown in Figure 4.25. Note that the default application name is based on the template name you choose in the project template dialog (see Figure 4.24).

Figure 4.25. Your custom template appears as a new project type under your development language.

[View full size image]

14.
To remove the template, navigate to your C:\Documents and Settings\billva\My Documents\Visual Studio 2005\Templates\ProjectTemplates and remove the .ZIP file located in this directory that corresponds to your custom project template.

Configuring Online or Local Help

I expect that you're going to need to use online (MSDN) or local help topics to explain issues that I don't cover here. The first time you press F1 or click on an item in the Help menu, Visual Studio prompts you to choose where you want to search for the help topics you need. You have three choices, as outlined in Figure 4.26. Note that while local help is faster, it's not nearly as complete as the online version. Note that this is a Standard or better feature.

Figure 4.26. Choose where help is sourced.

Previous Page
Next Page


Configuring Server (or Database) Explorers

Basically, the Visual Studio Server Explorer acts as a system-management console, permitting you to log on to servers (in the Servers list) and connect to data sources where you can manage databases and their components. In Standard, Professional, or Team editions, the Visual Studio 2005 Server Explorer has evolved quite a bit from the previous versions and is strikingly different than the Server explorer used in Visual Basic 6.0. Ah, before I forget to mention it, you'll have trouble finding the Server Explorer in the Express editions of Visual Studioyou'll find the Database Explorer appears in its place. As you'll also discover, the Database Explorer supports a subset of the Server Explorer features discussed in this chapter, so muddle along the best you can with this free Express versionI'll try to point out the features that are not implemented. However, remember that much of what the Server Explorer can do might still be possible by entering equivalent T-SQL into a query window in Visual Studio or by executing queries with the SQLCMD, SQL Enterprise Manager, or SQL Express Manager applications. A significant difference with the Express edition is the lack of .NET Data Providers. Out of the (free) box, it supports access to only a local instance of SQL Server Express and then only through an attached SQL Server database .MDF filebut the database file is not included; you'll have to create this yourself (I show you how in the next section). There is no facility to point to an external SQL Server instance in the Express edition from Visual Studio, but you can write applications to do so.

The Server Explorer permits you to browse the structure of a selected data source and create new objects on a selected data source. Its features are roughly the same for each backend to which you can connect. I'm going to focus on how it works with SQL Server, but you can still browse tables and create views and other elements against JET/Access, Oracle, DB2, or any other data source.

Creating a New Database with the Server Explorer

I assume that you're going to access one or more databases in the course of creating your applications. These can exist as stand-alone SQL Server database (.MDF) files, installed SQL Server databases on the local system, or on remote SQL Server(s) visible on your LAN or other data sources.

If necessary, the Standard, Pro, and Team versions of Visual Studio 2005 permit you to create new SQL Server (and just SQL Server) databases interactively with a number of techniques. While I prefer to use SQL Server Management Studio to create and administer databases, you can do at least part of the work from Visual Studio 2005. Note that SQL Server Express edition has its own version of the tool: SQL Server Management Studio Express. I discuss how to use both of these tools (at least to some extent) here and in Chapter 7.

The easiest way is to use the Visual Studio Server Explorer[6]. Open the Server Explorer (View | Server Explorer), right-click on the Data Connections icon, and choose "Create New SQL Server Database". This exposes the dialog shown in Figure 4.27.

[6] The Express edition assumes you're going to attach a preconfigured SQL Server database file.

Figure 4.27. Creating a new SQL Server database.

In this example, I point to the SQL Server located on the BETAV9 system and the SS2K5 SQL Server instance running on that machine. The new database is to be called "NewDatabase". Once the database is created, you can use the Server Explorer Database Diagram tools to create tables, relationships, and indexes.

Another approach (as suggested by the MSDN documentation) has you create a new "item" in a Visual Studio project. Let's walk through that approach, as it's the only way to create a new database in the Express Edition.

1.
Create a new Visual Studio project (Visual Basic .NET, C#, or Web project). I called the project "CreateNewDatabase".

2.
In the Solution Explorer, right-click the Project "CreateNewDatabase" and choose Add... | New Item and then SQL Database, as shown in Figure 4.28.

Figure 4.28. Use Add New Item | SQL Database to open create database dialog.

[View full size image]

3.
This dialog creates a new SQL Server .MDF database file and a Server Explorer (or Database Explorer) connection to access it. Note that this file is not attached to a SQL Server instance until it is opened. When the database is opened, it's attached (temporarily and invisibly) to the local SQL Express instance. This type of database is not seen by SQL Server Management Studio or other tools because ADO.NET and SQL Server create a user-unique instance to manage it. If you inspect the properties of this new database connection (see Figure 4.29), you'll see that the ConnectionString uses the "User Instance=True" option. Note that the database file is (currently) located in the application development directory. When deployed, this database file is copied to the user directory on the target system. I discuss this technique of database management in Chapter 9.

Figure 4.29. The ConnectionString generated by the Add New Item | SQL Database dialog.

[View full size image]

4.
Visual Studio next opens the Data Source configuration wizard that would normally permit you to select the database objects to include in your application dataset. However, since this new database has no user Tables or stored procedures, it does not make sense to continue past this dialog. I discuss the Data Source wizard in detail in Chapter 6. This approach adds the .mdf file to your project.

Creating a New Database in Code

Creating a new database is pretty simple to do in code[7], so Express Edition users can use this approach to create a new database. For example, if you execute the following T-SQL batch, SQL Server creates a new "mytest" database (see Figure 4.30).

[7] See SQL Server Books Online or online SQL Server help and search for "CREATE DATABASE".

Figure 4.30. Creating a new "mytest" database.

This T-SQL can also be executed directly in the Standard, Professional, and Team versions by using the Run Selection option of the stored procedure editor. I discuss this technique later in this chapter.


Creating and Managing Database Connections

The Server/Database[8] Explorer is the gateway to the data sources (databases) your application needs to access. This means you'll need to create a "Database Connection" to the SQL Server instances and specific databases or other databases. Each Database Connection refers to a specific database and login credentials. Behind the scenes, it's persisted as a unique connection string. You won't be able to create duplicate Database Connections, but you can connect to the same database using different credentials.

[8] As I progress through this chapter, I'll refer to the "Server" Explorer instead of Server/Database Explorer. I'll try to point out features not supported in the Database Explorer.

I've already shown how you can create a local file-based (private) Database Connection by adding a "SQL Database" item to your project, so let's walk through the process of creating a new Database Connection that addresses a remote or shared instance of SQL Server on your local system. No, you can't do this with the Express edition.

1.
Expose the Server Explorer by clicking View | Server Explorer. Initially, there won't be much there, as shown in Figure 4.31. I need to point to a specific data source so the explorer can drill down into its contents. Each "Data Connection" points to one (and only one) database located on the specified data provider (data source). You'll be able to create as many data connections as you needeach pointing to a different database.

Figure 4.31. The unconfigured Server Explorer (Professional and Team editions).

Each Data Connection contains a unique connection string that can be altered later (if necessary) to address another database, use different credentials, or set additional connection options. Depending on the rights granted to this connection, you might be able to alter the database structure (add/change/remove tables, stored procedures, views, etc.) using the Server Explorer. I'll show you how later in this section.

The Data Connection plays an important role in all of the Visual Studio 2005 wizards. Each time you're asked to provide a ConnectionString, Visual Studio returns a list of the current Data Connections to choose fromthis can save you a lot of time.

The "Servers" icon is not exposed in the Standard or Express editions. I discuss this feature later in this chapter.

2.
Since you need to point to a data source to activate the Server Explorer, right-click on the "Data Connections" icon and choose "Add Connection". You should now see the "Choose Data Source" dialog, shown in Figure 4.32.

Figure 4.32. Choose a Data Source provider.

This list is reduced to two items in the Express edition: SQL Server and Access database providers. You won't be able to set up connections to specific local or remote SQL Server instancesjust to the SQLEXPRESS instance via a privately attached (User Instance) .MDF file or a JET (.MDB) file.

Choose the data provider[9] that makes sense for this connection. If you plan to attach an existing .MDF SQL Server database file, choose "Microsoft SQL Server Database File". I discuss the implications of this approach in Chapter 9. For an existing server instance (and database), choose "Microsoft SQL Server".

[9] The Express editions expose only two providers: one for SQL Server database files and another for Access/JET database files.

3.
The next dialog you see (see Figure 4.33) should be familiar by nowyou worked with it when you created the ConnectionString for your application Template. Fill in the machine and instance name ("Server name") and authentication credentials ("Log on to the server"), and choose an initial catalog ("Select or enter a database name:") from the drop-down list. If the dialog can't connect to the selected server instance, you'll get an exception. While you can enter a default database, if the credentials or other issues prevent you from connecting, you're wasting your time. As I've said before, connecting can be one of your toughest hurdles, so you might find it necessary to review Chapter 9 before going on.

Figure 4.33. Using the Add Connection dialog to build a ConnectionString.

[View full size image]

Before you can choose SQL Server Authentication, your SQL Server instance must be configured to accept SQL Server credentials. By default, SQL Server instances are configured to accept only Windows authentication (SSPI) credentials. This means the credentials you used when logging on to Windows are used to gain access to the SQL Server instance. I also discuss how to configure SQL Server instance options in Chapter 9.

4.
Click on the "Advanced" button. This exposes the Advanced Properties property page dialog (see Figure 4.34). A number of important properties should be set here. Again, I discuss each of these properties in Chapter 9, so let's set some of these to give you an idea of how to set them:

  • Application Name: I set the Application name to make it easier to find references to this connection in the SQL Server Profiler. It's an essential tool to determine what SQL Server is being asked to execute and to debug all kinds of SQL Server performance and functionality issues. I expect to write an EBook on use of the Profiler at a later date. Check the support Web site for details.

  • ConnectionTimeout: I set this to 5 seconds, as most LAN-based connections can connect in less than 2 seconds. Setting this to a higher number just wastes your timeof course, if you're being paid by the hour....

  • Enlist: I set this to False because only a few of our connections used in Visual Studio require the use of transactions.

  • Max Pool Size: I set this to 10 to trigger pool overflow much earlier as I test prototype applications.

    Before you click "OK" on the Advanced Properties dialog, note that your configured ConnectionString is shown in the (editable) TextBox at the bottom of the dialog. You can try to copy from this TextBox if you want to use the ConnectionString in your code, but (by default) if you are using SQL Server credentials and did not specify "Save Password" in the Add Connection dialog (see Figure 4.33), the password is not exposed here.

    Click the Test Connection button in the Add Connection dialog when you're ready to verify that your ConnectionString is correct. Actually, if the drop-down of known databases works, it's not really necessary to test the connectionbut go ahead if it makes you feel more secure.

Figure 4.34. Setting ConnectionString properties in the Advanced Properties property page dialog.

5.
Once you feel comfortable with the settings, click "OK". This creates a new Data Connection in the Server Explorer, as shown in Figure 4.35.

Figure 4.35. The Server Explorer with a new Data Connection.

You're now ready to explore the schema of the database addressed by this new Data Connection. Note that when you first start Visual Studio, the Data Connections you have installed are not opened. Unopened Data Connections are shown with a small red X next to the icon, as shown in Figure 4.36. However, once connected, these Data Connections hold a connection open until Visual Studio shuts down. To open the Data Connection, click on the "+" next to your Data Connection.

Figure 4.36. Data Connections do not open when Visual Studio first starts.

After a brief wait, the Data Connection is opened and the Server Explorer is populated with the objects that you can browse and edit (in some cases). As shown in Figure 4.37, the Server Explorer exposes lists of objects, including Tables, Views, Stored Procedures, and other objects.

Figure 4.37. Setting Data Connection options.

A new feature for Visual Studio 2005 (and SQL Server 2005) permits developers to view the objects in the Server (and Database) explorer by object type or by "schema". By default, objects are listed as in previous versionsby their type. If you right-click on a Data Connection (as shown in Figure 4.38), you can choose to view the objects by schema. As I discussed earlier, a schema is not just the way that the tables are laid out in a relational model; it's also (in SQL Server 2005) how the objects are owned. That is, given the Server Explorer options, you can choose to view the objects exposed by the Server Explorer by the user (Schema) or the object that owns them.

Figure 4.38. Choose how the objects are displayed in the Server Explorer (Team Edition).

Once you choose the schema view, the list of objects visible from the connection is a function of the schema chosen. Choose the schema view (as shown in Figure 4.38) and drill into the list of Schemas by clicking on the "+" next to the Schemas icon. Next, choose a schema (I chose "dbo"), and a list of all of the Tables, Views, etc. visible to this selected schema is listed (as in Figure 4.39).

Figure 4.39. Drilling into the objects exposed to a specific schema.

[View full size image]

Note that the Data Connection menu also permits you to modify the existing Data Connection, as well as set several options (as shown in Figure 4.38):

  • Refresh: This tells Visual Studio to requery the connection to refresh the object lists. This might be needed when you add, change, or delete the objects (as when adding new stored procedures or tables).

  • Delete: This removes the connection from the list of registered Data Connections. This has no impact on existing databasesattached databases remain registered in the server's master database. Note that Visual Studio won't let you create duplicate Data Connections, but you can have several connections pointing to the same server and databaseit's just that the ConnectionString must be different for each one.

  • Modify Connection: This opens the dialog (similar to Figure 4.33) that permits you to alter the ConnectionString.

  • Close the Data Connection: This might be necessary to release the connection held by the connection pool. Note that closing the Data Connection does not close the server connectionat least, not immediately. You'll have to wait from 4 to 8 minutes for the connection pooler to release it.

  • New Query: This opens the Query Designer, as discussed later in this chapter. This tool is helpful when you need to build a block of T-SQL or more closely inspect the contents of a database object.

  • Detach Database: This detaches the selected SQL Server database from the local SQL Server Express instance (Express edition only).

  • Application Debugging: Permits application debugging using the Team system (Team version only).

  • Allow SQL/CLR Debugging: This option enables applications to set up CLR debug sessions with this connection. I discuss CLR debugging in Chapter 13 (Team version only).

  • Properties: This exposes the Data Connection property page, shown in Figure 4.40. Note that these properties are read-only but can be copied to extract an important property like the ConnectionString. Note that this Data Connection is marked as not "Case Sensitive". This attribute is set by the default case of the server. To change the ConnectionString property, click on "Modify Connection".

    Figure 4.40. The (read-only) Data Connection properties page.

Previous Page
Next Page


Managing Database Objects with the Server Explorer

We're ready to explore the Server Explorer a bit deeper, so let's talk about each of the object lists with which you can work:

  • Database Diagrams: This feature (which was going to be dropped from Visual Studio) permits you to view and manage the database tables and relationships. That is, you'll be able to set up as many database diagrams as needed with whatever tables you deem appropriate. The relationships between these tables can be shown as well as edited. Let's walk through the process of creating a new Database DiagramI know you'll find this a useful side trip.

The first time you reference Database Diagrams for a specific database, Visual Studio exposes a dialog complaining that the objects (tables and stored procedures) needed to support diagrams are not present. If you want to support diagrams in your database, you'll need to let SQL Server create these objects. In SQL Server 2005, you won't (necessarily) see these objects, as they are added as system stored procedures and tables.

1.
Right-click on the Database Diagrams icon and choose "Add New Diagram". This opens a "blank slate" diagram pane on which you can add database tables for your diagram (as shown in Figure 4.41). I'm going to select a few related tables for the diagram. Later I'll add a new database table. Note that I switched back to the more familiar "Object" (or Default) view.

Figure 4.41. Choosing "Add New Diagram" opens the Add Table dialog on a new diagram pane.

[View full size image]

2.
Select as many tables as you want to appear in the diagram from the (multi-select) list and click "Add". When you're done adding tables, click "Close". The selected tables are then drawn on your new diagram along with any existing primary key/foreign key relationships. On closer inspection, you can see the columns designated as primary key (the small key) and the one-to-many relationships between tablesthe "one" side is a key and the "many" side is shown as an infinity ( [image]) sign. You can zoom in or out on the diagram and align the tables anywhere you choose.

When you create new database objects (tables, stored procedures, views, etc.), the Visual Studio (and SQL Server) tools don't always recognize them without refreshing the list of objects. Right-click the list and choose "Refresh" to get a current list.

3.
Each of these tables (and relationships) is editable. If you right-click in the whitespace below the column list, a dialog (shown in Figure 4.43) opens that permits you to edit the data table, set or remove a primary key, add or remove columns, or manage the diagram's relationships.

4.
You can also use this menu to manage the indexes created for the selected database table or manage the check constraints. If you choose "Indexes/Keys", the dialog shown in Figure 4.44 is exposed. Here, you can see existing indexes and add or delete indexes as needed.

Figure 4.42. Tables added to diagram are shown with existing relationships.

[View full size image]

Figure 4.43. Right-click to edit the data table or relationships.

[View full size image]

Figure 4.44. Clicking on "Indexes/Keys..." opens this dialog.

[View full size image]

5.
In a similar way, you can edit the relationships established for the database tables shown in the diagram. Right-clicking one of the relationship lines (as shown in 4.43) permits you to delete the relationship or view its properties in the Visual Studio properties dialog.

Note that none of the changes you make to the diagram is persisted to the database until you save the diagram.

6.
Let's add a new table to the database (and the diagram) and add a relationship to the existing tables. Right-click on the database diagram (in the whitespace) and choose "New Table..." to create a new table (as shown in Figure 4.45).

Figure 4.45. Create a new table or select from the list of existing tables.

7.
To add this new database table (which is what you're going to want to do when creating a new database), enter a name for the new table in the "Choose Name" dialog. Note that this table is not immediately added to the databaseit won't be until you save the database diagram. I'm naming this table "AuthorDetails". I generally make table names plural, as they are collections of rows of like kind.

IMHO

I assume that you have rights on the database to add new tablesyour DBA might not (should not) have granted these rights.

8.
The new database table dialog created when you choose "Add New Table" permits you to fill in the column details for the database table you're creating. The most important properties of the column must be set in the Visual Studio Properties dialogall you get to provide in the new table dialog is the column name, data type, and allow nulls. Without going into a long spiel about suitable data types, just consider that while disk space is much cheaper nowadays, the more you put in memory (the larger the datatype), the fewer the rows you can cache. I also recommend staying away from fixed-types (Char and Nchar) except for cases when the data is always (as in, "always") the same size (as in a 2-byte State code). Be sure to mark fields as permitting NULL if you know there are going to be cases where the data is not yet available (as in DateRetired). I discuss how to build data tables in Chapter 2, "How Does SQL Server Work?".

Take a look at Figure 4.46. Note that I've defined a number of columns and set the datatype and nullability for each. No, I didn't have to remember any datatype namesthese were provided in a drop-down list as I typed. Note that the column names have no spaces. While this is permitted in SQL Server syntax, it's not something you want to dotrust us.

Figure 4.46. Add New Table dialog.

9.
We're not done yet defining the database table. We still need to set specific properties on the Au_ID column. In this case, I need to use the Visual Studio Properties dialog (shown in Figure 4.47) once I click on a specific column. Select the Au_ID column and note that the Properties dialog shows the column properties. I've set several properties on this and the other columns in the new database table. These properties include:

  • Description: This can be used behind the scenes as a column heading. In any case, I set it to document the purpose of the column.

  • Identity Specification: These settings determine that this column is an "Identity" column of some kind. No, one should not set the AuthorDetails..Au_Id column as an Identity because it's the "many" (the Foreign key) side of the relationship. That is, the Au_ID column must be set to an existing (valid) Au_ID value in the Authors table to reference an existing author. This means the AuthorDetails table is a "child" tablethe Authors table is a "Parent".

    If you're creating a "Parent" table, it often makes sense to configure the primary key to be auto-populated as an Identity. Once you change "(Is Identity)" to Yes, the Identity Increment and Identify Seed are set to 1. This means that SQL Server writes a guaranteed unique integer to this column when new rows are added to the database. Depending on the size of the table you expect to build, you might consider changing the datatype from int to bigint or smallint. As I discuss in Chapter 12, "Managing Updates", where I talk about concurrency, you can use other methods to uniquely identify each row, but setting up an Identity is easiest. As an alternative, you can to set your column datatype to "Uniqueidentifier" if you wish to generate a GUID to make each row unique, or you can create a custom primary key value that does the job.

  • Default Value or Binding: While I don't want to set a default value for the primary key, I would generally set a default value on any columns in the table (except the TimeStamp) that should have a non-null value.

Figure 4.47. The Visual Studio Properties dialog when focused on a specific table column.

[View full size image]

10.
The next step is to set a primary key (PK) for the new database table. Remember, the PK is really a requirement for any relational table. While not absolutely necessary, without a PK, ADO.NET won't be able to identify specific rows to updateand neither will you (at least, not easily). A PK can consist of one or several database table columns taken together to return a unique value. For example, you could use a customer's name as a unique identifier if you have only a few people to trackthat is, until you get another customer with the same name. People have tried to use Social Security Account Numbers (SSAN), but these are not uniquethere are too many "duplicates" out there.

Any column(s) you designate as PK must contain a unique (non-NULL) value. This means as you add rows to the table, the new PK value can't exist in the table beforehand.

Once you decide which column(s) constitutes the PK, left-click to select the column and right-click to open the column operations menu, as shown in Figure 4.43. Choose "Set Primary Key" or "Remove Primary Key" to change the PK column.

11.
To permit SQL Server to easily locate rows in this table and improve query performance, you'll want to create one or more indexes for your table. This is fairly easysimply right-click on the database table and choose "Indexes/Keys". When you designate a PK, Visual Studio automatically generates an index based on the PK columns, as shown in Figure 4.48.

Figure 4.48. Creating an Index for your database table.

[View full size image]

If necessary, click "Add" to create a new indexyou'll need to give it a unique nameor "Delete" to remove an existing index. Sure, you might find it necessary to create several indexes (especially for large tables that are searched in a variety of ways). For example, you might want to regularly locate author details on city or on retirement date. In this case, it might (just might) make sense to add indexes on these columns.

Consider that each index slows down the INSERT and UPDATE operations, as they must be updated along with the data rows. Indexes also take more space in the RAM cache and on disk. Use the tools in SQL Enterprise Studio to analyze and optimize performance against your table and queries once they are populated with data.

12.
No, we're not quite done. The new AuthorDetails table has not been related to the existing Authors table. This means that if I add new rows to the AuthorDetails table, they might not have valid Au_ID values. Adding relationships is also easy. You have a couple of options here:

  • Use the drag-and-drop technique: In this case, simply select the primary key column (Au_ID) in the primary key (parent) table (Authors, in this case), drag the arrow to the foreign key (child) table (AuthorDetails, in this case), and drop on the PK of the child table (AU_Id). This opens the Tables and Columns dialog, as shown in Figure 4.49.

    Figure 4.49. Creating a PK/FK relationship.

    When you click OK, the Foreign Key Relationship dialog is exposed (see Figure 4.50). This dialog names the new foreign key and permits you to make whatever configuration settings you feel are necessary. Here you specify:

  • How foreign key table rows are handled if rows from the parent are deleted. For example, you can have SQL Server make sure that if a parent row is dropped, all related child rows are dropped as wellthe default is "No action" on both deletes and updates.

  • These options also permit SQL Server to ensure that no row is added to this foreign key (child) table if a valid parent row is not already in the database.

Figure 4.50. Setting relationship options in the Foreign Key Relationship dialog.

[View full size image]

13.
At this point, nothing has been committed to the database, so if you close the Database Diagram tab, you'll be prompted to save changes to the Diagram (which will save the changes to the database). But before you do that, you need to save your work in another way. One of the most powerful features of this tool is the ability to "script" the changes you're about to make to the databaseat least, as far as adding and changing tables, indexes, and relationships are concerned.

Right-click on the new database table we've been working on in the Database Diagram and choose "Generate change script...". This exposes yet another dialog (see Figure 4.51) that shows the T-SQL about to be executed to carry out the change you've requested. You might find it useful to save this to a file and review it with your DBA.

Figure 4.51. The change script generated by Visual Studio.

14.
Okay, we're now ready to save (and commit) our Database Diagram. Click File | Save. At this point, you'll be given an opportunity to name your new Database Diagram. Once you provide a name, you're prompted (see Figure 4.52) to confirm that you're ready to accept changes to all affected database tables. Ah, if you click "Save Text File", all you'll get is a file that contains the names of the database tables to be changednot particularly useful.

Figure 4.52. Confirm that you want to change the indicated tables.

15.
Once you click "Yes", the change script is executed and the new objects (including the Database Diagram) are stored in the database for everyone to seeat least, everyone with rights to the database. Upon inspection of the Server Explorer, you'll see that our new table "AuthorDetails" is now listed among the other tables (see Figure 4.53). If it's not, you might have to right-click on the Tables icon and choose "Refresh". Note that by clicking the "+", you can drill down to the list of columns. As you select a table or column, the Visual Studio properties dialog is populated with the property settings for the selected item. This is a good way to determine the datatype for a selected column or simply determine the approximate number of rows in the database table.

Figure 4.53. The new AuthorDetails table is now listed under "Tables" in the Server Explorer.

Changing Existing Objects

Once your database tables, relationships, and indexes are built, you're going to want to edit these items from time to time. This is not at all difficult, as the Server Explorer is fairly adept at these operations.

Managing Database Tables

To change an existing database table, right-click on the table and choose "Open Table Definition". This opens a new Visual Studio tab (see Figure 4.54) that includes two dialogssimilar to those used when I created tables with the Database Designer.

Figure 4.54. The Open Table Definition dialog.

Adding or changing data in an existing table is also possible with the Server Explorer. The Visual Studio 2005 implementation is rich with features that should make this easier (and safer) than ever. Let's walk through the process of adding a few rows to the AuthorDetails table.

1.
Right-click on the AuthorDetails table in the Server Explorer and choose "Show Table Data". You should see a dialog similar to Figure 4.55.

Figure 4.55. The "Show Table Data" dialog ready to accept new rows.

[View full size image]

Sadly, the UI here in Visual Studio 2005 is different than that used in the "companion" SQL Server Management Studio. The SQL Server tools support all of these same operations, but they use different names for the tasksit's almost like the two groups weren't in the same building working for the same boss.

2.
At this point, you need to enter data in each of the columns (that require an entry). Remember that the AuthorDetails Au_ID column is a foreign key that's tied to a valid (existing) author, so you'll need to enter an appropriate Au_ID value here. As you enter data, the dialog prompts you to indicate that individual columns are being changed. New popup dialogs (see Figure 4.56) show that the data has changed, that it has not been committed, and the original value (before the change).

Figure 4.56. As data cells are changed, the error provider shows additional information.

3.
When you finish filling in the (required) columns and navigate to another row or click File | Save[10], the row is committed to the databaseas long as the PK/FK and other constraints are not violated.

[10] Or click the "File Save" icon.

Unlike Visual Studio 2003, you won't be able to drag Tables, Views, or stored procedures from the Server Explorer IDE to a Windows Form. That functionality has been moved to the new Data Source paradigm, as I discuss in Chapter 7, "The Visual Studio 2005 Data Tools". However, if you're building an ASP.NET page, the Server Explorer is still the only way to drag and drop data elements to the design surface.


Managing Queries with the Query Designer

One of the issues I've had with earlier versions of Visual Studio is the inability to create SQL queries to incorporate the T-SQL in an ad hoc query or to simply query the database to sample the data or test performance. While Microsoft has incorporated the Query Designer in a number of earlier versions (clear back to earlier versions of Visual Basic), it was not always easy to find. In Visual Studio 2005, the Query Designer can be addressed from a number of places. You'll be able to get to it when customizing a Visual Studio Data Source, creating the SQL for a stored procedure, creating a "DataSet" for a BI report, and here in the Server Explorer.

IMHO

Yes, the Query Designer used to be called the Query Builder. Tomorrow, there's no way to predict what it will be called.

I expect that you might be familiar with the Query Designer by now, and I don't want to kill any more trees than absolutely necessary as I teach you how to use it. However, I do want to discuss a few new features that you'll find useful and a few other (sinister) issues[11] that might bite you in the butt as you walk by. Let's walk through a number of typical scenarios.

[11] One of these issues was pointed out to Peter as he worked on a Reporting Services dataset. His wife saw a mistake as she passed by (vacuuming the room).

Getting Familiar with the Query Designer

Navigate to the Server Explorer, right-click the Table (or View) icon, and choose "New Query". This opens a four-pane tabbed window (see Figure 4.57) and a dialog that asks which tables should be included in the query.

Figure 4.57. Setting up the Query Designer.

[View full size image]

Here's your first trick: If you want to create a query window and don't really care about the graphical part, click "Close" on the Add Table dialog without selecting any tables. This opens a blank "diagram" (top) window (what I call the table column selector). Use the pane selectors to turn off all of the windows except the SQL and query results windows. At this point, you can enter a query in the SQL window, and while the Query Designer might complain that it can't create a graphical representation of your query, clicking OK executes the query anyway and returns the rows affected or rowset in the query results window.

The Query Designer Panes

Note that the four panes can be toggled on and off with the pane selectors, the Query Designer menu, or keyboard shortcuts. As you modify each pane, the Query Designer attempts to build a SQL query and change the other panes to match these changes. For example, as you select columns in a table column selector (the top pane that Microsoft calls the "Diagram" pane), the SELECT statement is coded to include these columns and the "criteria" pane (the second pane from the top) reflects these changes as well. The columns are placed in the SELECT clause and the "criteria" pane based on the order in which they are checked off in the column selector.

If you choose more than one table to include in the query in the "Diagram" pane (from the Table selector), the Query Designer creates appropriate (and sometimes inappropriate) JOIN syntax to return the SQL product. This is the issue that Peter's wife pointed out. Because the Query Designer creates one or more JOIN statements for each possible relationship, if you aren't careful, it can get a bit too aggressive and create JOIN statements that don't really make sense.

The Criteria Pane

The "criteria" (second) pane also determines which columns are included and aliased, and which order they appear in the SELECT. This pane also permits you to set the ORDER BY clause as well as the WHERE clause criteriaup to a point. To change the order in which rows are returned, set the "Sort Type" for the column(s) that you wish to sort by. You can change the sequence of this sort by setting the Sort Order. For example, you can sort on Year_Born and then by Author name. The Filter criteria column is used to generate the WHERE clause criteria. This permits you to specify criteria that must be met for each column for which you need to test. For example, you can set the Filter on the Year_Born column to accept only authors born in1947 (a good year for authors). The Filter criteria also includes several "Or" columns that can be used to add filters to your WHERE clause. These settings are shown in Figure 4.58.

Figure 4.58. Configuring the WHERE and ORDER BY clauses with Sort and Filter.

Eventually, you'll discover that the Query Designer has indeterminate limits, so as your query gains in complexity, you'll find that the generated SQL stops working as expected.

The SQL Pane

The "SQL" query pane (third from the top) shows the generated SQL. The SQL dialect used matches that of the selected data provider. Yes, you can directly edit the generated SQL, but if you make a mistake (or the Query Designer simply can't understand the SQL), the Query Designer returns an error. I often find it handy to copy and paste from the SQL pane when I need to build a SQL string to paste directly into a prototype query in our code.

The query results pane (bottom pane) shows the results of the query after you click the "Execute Query" button (see Figure 4.59). This pane also permits you to change the database table in question if the query addresses a single table.

Figure 4.59. The Query Designer toolbar.

You can also validate your hand-coded SQL by clicking the Test SQL Syntax button. You can also incorporate a GROUP BY clause or a derived table in the T-SQL by clicking the appropriate buttons in the Query Designer toolbar.

What's a "Derived" Table?

One of the other options in the Query Designer toolbar is "Add Derived Table". Simply put, a "derived" table is not unlike what I call a "subquery". It's a SELECT statement that returns an in-memory rowset referenced as a database table or VIEW in a query. Derived tables are saved (only) in memory and not persisted to the database. Think of a derived table as a temporary rowset that lives for the duration of the query. This approach can simplify your query by permitting you to gather rows to join in a separate phrase. For example, the following query (see Figure 4.60) uses a derived table to return a rowset from the TitleAuthor table and use it to JOIN with the Authors table.

Figure 4.60. Using a derived table.

Changing the Query Type

Sure, I expect that only some of the queries you'll want to generate are SELECT queries. The Query Designer is capable (to an extent) of creating action commands as well. Let's walk through these options, as you'll find them useful in many circumstances as you manage your database. The Change Type drop-down (see Figure 4.61) permits you to select one of six options. Note that the Filter and other criteria settings are made available in these other query types where it makes sense.

Figure 4.61. The Change Type options for the Query Designer.

  • Select: The "Select" query type generates a SELECT query, which I've already discussed. However, we need to revisit this for a minute, as I didn't talk about how to change the JOIN behavior. When you drag two or more tables to the diagram pane, the Query Designer scans the tables looking for matching column names. If it finds a match, it codes an SQL JOIN to interrelate the tables. For example, if I drag the Titles and Title_Author tables to the diagram pane, the Query Designer creates an INNER JOIN between the ISBN columns, as shown in Figure 4.62.

    Figure 4.62. The Query Designer creates an INNER JOIN by default on matching columns.

    Okay, but what if that's not the type of JOIN you want to use between the two tables? Well, the Query Designer permits you to alter the SQL JOIN generated by a couple of techniques.

    1.
    First, hover over the INNER JOIN symbol (as shown in Figure 4.63). This displays the current JOIN configuration.

    Figure 4.63. The Query Designer showing how the JOIN is configured.

    2.
    Next, right-click the join graphic. This opens a selection dialog that lets you choose the type of JOIN to generate, as shown in Figure 4.64.

    Figure 4.64. Choose the type of JOIN from the selection dialog.

    By using the GUI interface in the Query Designer you can choose whatever JOIN makes sense for your query. However, there is yet another way to set up the JOIN.

    3.
    Choose "Properties" from the JOIN-type dialog (shown in Figure 4.64). This opens the Visual Studio properties page for the Query Designer Join Condition and Type, as shown in Figure 4.65.

    Figure 4.65. The JOIN properties page.

    4.
    Click on the ellipsis (...) button to expose the JOIN configuration dialog, shown in Figure 4.66. From here, you can choose from a list of expressions to construct the JOIN. Again, it's up to you to decide what makes sense for your query. Any change you make here is reflected in the graphic diagram tab, as well as in the generated SQL in the SQL tab in the Query Designer.

    Figure 4.66. The JOIN configuration dialog.

    We're ready to move on to the other types of queries that the Query Designer can generate.

  • Insert Results...: This query type is useful to extract a subset of the rows from one table (or query JOIN product) and insert them into another (permanent) table. Once you select a source table, you can use the Insert Results query type to insert all or a subset of rows into another target table (assuming it has a compatible schema), as shown in Figure 4.67. You can also use this query type to insert rows from the product of a JOINsimply select multiple tables and make sure the JOIN syntax returns the appropriate rows and columns to match the target table. Setting up the Insert Results query is fairly easy:

    1.
    Choose the source table(s) using the Add Table dialog. These are added to the diagram pane.

    2.
    Choose appropriate columns in the source table(s) in the diagram pane.

    3.
    Once the Insert Results query type is selected, you're prompted for the target table. Make sure its schema matches that of the source table or JOIN product.

    4.
    Set any Filter criteria to generate a WHERE clause that returns only the desired rows.

    5.
    Add whatever columns you need to the criteria pane (non-output) to further filter the source rows.

    6.
    Add whatever TOP or other WHERE clause arguments are needed to limit the number of rows inserted into the target table.

    Figure 4.67. Insert Results Query type.

    [View full size image]

    In the example in Figure 4.70, I join three tables (Authors, Title_Author, and Titles) and return columns from Authors and Titles. I added a filter on Year_Published in the Filter pane to restrict the rows returned to just authors whose books were published in 1995. In a practical sense, this Filter value should probably be handled as a query parameter. When the Insert Results query is executed, Visual Studio returns a "rows affected" message.

    Okay, let's change the criteria for Year_Published to "@YearWanted" (see Figure 4.68). By preceding the filter value with an "@", I indicate that I want this value captured as a parameter and that the SQL is to be modified to accept this parameter at runtime.

    Figure 4.68. Using an input parameter in an Insert Results query.

    [View full size image]

    Now, when the query is executed, Visual Studio prompts us for the input parameter, as shown in Figure 4.69. Visual Studio remembers the value you used from the previous execution but defaults to NULL. You might want to phrase your query to "ignore" the input parameter if the value passed in the parameter is NULL or some special string.

    Figure 4.69. Visual Studio prompts for query parameters.

  • Insert Values: This query type generates a single INSERT statement, as shown in Figure 4.70. For this query type, you'll need to choose the columns that are to accept new values. Don't select any Identity or Timestamp columns, as the Query Builder blindly assumes you know what you're doing. Set the values for each of the columns selected in the (top) "diagram" pane. These values can be literals, dates, numbers, or expressions, but the value must match the datatype and size of the target datatype, as dictated by the table schema definition. This means that the dates must be valid and strings must be no longer than the string length defined for the target column, and the numbers you supply must conform to the range limits defined for the target datatype.

    Figure 4.70. The Insert query type.

    Unfortunately, someone at Microsoft was asleep when they wrote the Insert Values and Update query type Command Builder code. The problem occurs when you fill in the "criteria" pane with New Column values: The CommandBuilder routine that captures these values doesn't know how to deal with the "O'Malley" issue. For instance, if you enter the name "George O'Malley" as a New Value, the Query Designer strips the interior single quote instead of doubling it up (as it should). This means that if you don't catch it, Visual Studio creates an INSERT statement and sets the Author name to "George OMalley". Thankfully, George is used to this abusebut his wife is not, so you had best fix it before she finds out. When working with the Query Designer and Input Values or Update query types (and you have embedded special characters in a string literal), you'll need to "double" up the special characters to get them to work correctly. So to fix George, I need to enter his name as "George O''Malley"and don't forget about his wife.

    Nope, I don't expect you'll use the Insert Values query type oftenat least, not from the Visual Studio IDE. It's useful, however, to remind you of the correct INSERT query syntax so you can paste the (corrected) T-SQL into your application's ad hoc commands.

    As with the Insert Results query, you can set the criteria pane to prompt for input parametersthese values are fed to the INSERT statement at runtime. Most of these query types accept input parameters as placeholders for Filter criteria and some other values (such as the TOP operand that limits the number of rows returned by the query). Fortunately, the code to manage inbound parameters knows how to deal with the "O'Malley" issueso does ADO.NET (assuming you use a Command object with input parameters).

  • Update: This query type is used to generate a single T-SQL UPDATE statement. Let's step through the process of setting up an Update query type.

    1.
    Click the Table (or View) icon in the Server Explorer and choose "New Query".

    2.
    Select the database table to update.

    3.
    Choose the column(s) to change. Note that the PK column is not shown as updateable. As a general principle, you should never change the PK, as this violates fundamental relational database principles. If you need to change the PK, you should drop (delete) the row and add (insert) it with a new PK.

    4.
    In the criteria pane, choose the PK column from the drop-down list and set the Filter to select the row(s) that are to be updated. If you don't specify a set of rows to change, all rows are updatedgenerally, this is not desirable. Sure, you can also set the Filter value for one or more of the columns to be updated.

    In this case, I'm updating any row with the author name (Author) of "Fred Flintstone" and Year_Born = 100 or with a PK (Au_ID) of 99999. Note that when you enter more than one value in the Filter column, the SQL logic uses "AND" to combine the logic. If you want an alternative (OR) Filter, enter the value in one of the Or... columns. Check out the SQL in Figure 4.71. Note how the WHERE clause is constructed using the Filter expressions and how the AND and OR clauses are generated.

    Figure 4.71. Using more complex AND/OR logic in an Update query type.

    [View full size image]

    5.
    In the criteria pane, set New Value to the new value to be set for the specific row(s) targeted in the Filter criteria. Yes, these values can be set up to accept parameters as described earlier.

    6.
    Click the "Run" icon when you're ready to make the change.

  • Delete: The delete query type is designed to construct a simple DELETE SQL statement that (like the update query type) can delete a specific row or an entire set of rows. The questions you need to ask when deleting rows from your database table include:

    • Is the row you're deleting a parent of a parent/child (PK/FK) relationship? Remember, you should not delete parents that have children (FK-related rows). Yes, you can program SQL Server to "cascade" deletes so children are automatically deleted when parents are deleted.

    • Are you deleting all of the rows in a table? If so, it's probably faster to execute the T-SQL TRUNCATE TABLE command.

    • Do you want to be able to roll back this change? That is, is this delete part of another T-SQL operation that must succeed before the row(s) is deleted? If so, consider use of transactions that wrap the delete statement with the other interdependent (atomic) statements. Have you considered simply updating the database to mark "deleted" rows instead of removing them from the database? This might be faster to execute and leave the rows around so they can be "undeleted" later.

    In this example (see Figure 4.72), I simply changed the query type from the previous update query type example. Notice how the SQL is morphed to keep the WHERE clause but discard the SET statements.

    Figure 4.72. Configuring the delete query type.

  • Make Table...: When you want to create a new table from a rowset or product created by a SELECT query, you can choose the make table query type. In this case, you'll be prompted for the new target table. Nope, this query type won't work if the target table already existsyou'll need to use the insert results query type if the table already exists.

    I use this approach quite often to create temporary (#temp) tables that contain products that are pertinent to the current logic context. For example, when working with a large database, one could "pre-fetch" a set of pertinent data row keys that apply to the current user. These can be persisted in tempdb (in a Smart Client [Windows Forms]) "connected" application so when the client needs to execute a query, a JOIN can be made against the #temp table. This has been shown to dramatically improve performance in many situations.

    The problem with using #tempdb tables from Visual Studio is that, behind the scenes, Visual Studio opens a new connection to SQL Server and closes this connection as soon as the test query is executed. This flushes the #temp database tables so they can't be seen after the query is executed.

The Query Designera Recap

For the most part, the Query Designer can be a useful tool to help generate simple queries to test a concept or provide tested SQL to incorporate into an ad hoc query. It is fine as far as it goes. However, it won't be long before your skills outgrow its usefulness. The Query Designer is virtually useless for stored procedures or complex queries, as the GUI-generation code gets in the way. If you need a more sophisticated tool to test and debug queries of any type, I recommend using SQL Management Studio or the new SQL Express Database Management tools. SQL Management Studio also includes a query analyzer to tune the performance and resource footprint of your queriesit's a far cry from what Visual Studio 2005 affords. You might also find that SQLCMD (the replacement for OSQL and ISQL) command-line tool is handy.

One of the more irritating problems with the Query Designer is that these queries cannot be saved with a normal Visual Basic or C# project. You can save them only in a "Database" (.dtq) project. This is an issue that Microsoft wants to fixif they have time to spare after adding all of their new features.

Managing Database Views Using the Query Designer

Just as with database tables, you can use the Server Explorer to create and edit database views. Remember that an SQL view is simply a T-SQL query that's used like a macro in T-SQL queriesit can be used anywhere you can use a table. Views can be indexed and, in some cases, can be updateable. See Chapters 2 and 12 for more details. At this point, I want to (briefly) discuss creating views with the Server Explorer.

1.
Right-click the Views icon in the Server Explorer and choose "Add New View".

2.
Just as with the Query Designer select query type, add one or more tables (or views) to the diagram (top) pane. If these tables are related, the Query Designer automatically tries to create appropriate JOIN syntax. Note that if there is more than one possible way to JOIN the tables, all are included in the JOIN clauses created. This might not be what you have in mind.

3.
Set the Alias column in the criteria pane to rename any columns generated from expressions or with unsuitable column names.

4.
As with the other query types discussed earlier in this chapter, set the Filter and Sort columns in the criteria pane to define the WHERE and ORDER BY clauses to filter and sort the view's rowset.

5.
Add whatever TOP or other expressions to the SELECT statement to further restrict the number of rows returned or to expose the rowset in a special wayunique to this view. In the example in Figure 4.73, I set the collating sequence to return a case-sensitive view of the author's table. This means when I SELECT against this view, I can differentiate between "MacDonald" and "MACDonald"if ever that becomes an issue.

Figure 4.73. Creating a view with a special collation sequence.

Previous Page
Next Page


Using the Server Explorer to View Synonyms

One of the more useful (and unsung) features of SQL Server 2005 is the ability to define Synonymswhile the Server Explorer can't create Synonyms, it can view the Synonyms defined in a selected database. Apparently, Oracle has supported Synonyms for some time, but this feature is more than just another "check-off" item. Synonyms permit developers to create an abstract layer over schema-owned database objects like Tables, Views, stored procedures, and user-defined and CLR Functions.

In earlier versions of SQL Server, developers often created Views to alias Tables, but this approach does not work for Functions or stored procedures. No, Synonyms can't be used to alias lower-level objects (like columns) or higher-level objects like databases, but a Synonym can be used to dereference (alias) local SQL Server objects. You can dereference objects on remote SQL Servers as wellassuming you've already set up the remote server as a linked server. For example, to reference a remote database table, you can set up the linked server (a one-time operation) and code a T-SQL query to reference the remote SQL Server database Table (as shown in Figure 4.74).

Figure 4.74. Add a linked server and reference it in a SELECT query.

[View full size image]

In the query, the FROM clause references the Authors table on a remote (linked) server. To make coding easier, you can create a Synonym to address the linked server table, as shown in Figure 4.75.

Figure 4.75. Creating a Synonym to address a linked server's table.

[View full size image]

Once the Synonym is created, you can code your T-SQL to reference the linked server's table via the Synonym (as shown in Figure 4.76).

Figure 4.76. Referencing a table via its Synonym.

Sure, you can create Synonyms for other objects, such as stored procedures, user-defined (and CLR) Functions, and Views.


Using the Server Explorer to Manage Types

SQL Server user-defined types have been around for quite some timesince the earliest versions of Microsoft SQL Server. These aren't really user-defined system types as much as aliases to system data types. That is, a SQL Server user-defined type (UDT) simply dereferences an existing SQL Server system data type. Sure, the Server Explorer can manage user-defined data typesbut only the new Common Language Runtime (CLR) data types. It does not see the more-familiar "aliased" types I've been using for decades. I think this is an obvious oversight.

A lot of attention has been drawn to CLR user-defined types in SQL Server 2005; I'm going to discuss this new approach to handling (very) special data typing in Chapter 13, "Managing SQL Server CLR Executables."

Because the Server Explorer does not help with SQL Server UDTs, you'll have to use the SQL Server Management Studio to manage them. You'll have to drill pretty deep to get to themthey're under <database> | Programmability | Types | User-defined Data Types. SQL Server Management Studio provides an easy-to-use dialog to create SQL Server UDTs, as well as a way to define (bind) a Default and Rule to the type (as shown in Figure 4.77).

Figure 4.77. Using the "New User-Defined Data Type" dialog to define a UDT.

[View full size image]

You can save the UDT definition to a T-SQL script by clicking the "Script" button. Notice that I bound the new UDT to a Rule. Once you define a SQL Server user-defined type (as opposed to a CLR type), you can create a SQL Server Rule that applies to all uses of the type. For example, suppose you create a new "postal code" data type (I used SQL Server Management Studio to script an existing type to a new query window, as shown in Figure 4.78).

Figure 4.78. Creating a new user-defined SQL Server data type.

[View full size image]

Note how the new type (dbo.PostalCode) is simply defined as a varchar(10). Once defined, I can create a SQL Server Rule to apply to this type. This Rule is applied each time data is added or changed in the column defined with this datatype. While you don't need to define a user-defined type (UDT) to apply a Rule, doing so means that you can change the Rule and have the change apply universally over the entire databasewherever the UDT is used to define the data type for a column. For example, our PostalCode data type has a formatting rule that dictates how the data in any column whose data type is PostalCodeit's shown in Figure 4.79.

Figure 4.79. Defining a Rule to dictate how U.S. postal codes are to be formatted.

[View full size image]

IMHO

Ah, T-SQL Rules are not really supported by Microsoft, but keep reading. No, it's not like when politicians ignore rules....

Once the SQL Server Rule is defined, you have to bind the Rule to the column(s) or UDTs where it's to be applied. In this case, I want to bind (apply) this Rule to the ZipCode column of the Biblio.dbo.ZipCodeState table. I set up this table using the Table Designer, as shown in Figure 4.80.

Figure 4.80. Defining the ZipCodeState table.

Notice that the data type selected in the Table Designer was selected from a list of valid data typesthis list includes both SQL Server UDTs and CLR UDTs. I'll show you how to define these CLR UDTs in Chapter 13.

Binding a SQL Server Rule is not hardyou simply call the sp_bindrule system stored procedure and pass the rule and target, as shown in Figure 4.81.

Figure 4.81. Binding a SQL Server Rule to a SQL Server UDT.

From this point forward, whenever data is inserted or updated in any column whose data type is set to PostalCode, the Rule code is executed to validate the incoming value. If the data does not conform to the Rule criteria (if I may leverage that word), the insert or update is rejected and SQL Server returns an error. Of course, this means that if your application accepts postal codes from outside the U.S., you'll want to redefine your Rule to include other formatsor use another approach.

Before you run off and start coding and binding Rules to UDTs, be aware that Microsoft no long likes this approach (for some reason). The engineers up on the hill in Redmond (and Bangalore) have already deprecated Rules in the SQL Server Management Studio IDE, so you can't create new Rules with a templateas you can with other objects. For some reason, Microsoft wants developers to use Constraints instead. Unfortunately, Constraints can't be bound to UDTs. This means you'll have to create a Constraint and bind it to each individual column where it's to be appliedwhat a PIA (and I don't mean Primary Interop Assembly)! I'm also of the opinion that Microsoft will have to continue to support Rules foreverat least, in a "compatibility mode". If they don't, they're pooched, and so are a lot of SQL Server customers.


Using the Server Explorer to Manage Assemblies

Again, a lot of emphasis has been placed on the ability to create Common Language Runtime (CLR) executables that can be called from within SQL Server. These executables can be in a number of forms, including stored procedures, functions, user-defined types, and aggregates. I devote Chapter 13 to the discussion of this new type of executable and where they make sense.

For now, I just want to mention (in passing) that the Server Explorer can expose paths to the files used to build a specific UDTassuming you have not moved the files. I'll discuss how to use the Server Explorer to navigate to these files and launch the projects used to build them in Chapter 5.


Using the Server Explorer to Manage Servers

Another important role the Server Explorer can play is as a tool to view and manage a number of aspects of a registered server.

In earlier versions, the IDE enumerated the servers it could see on the LAN so that you could choose from a server to manager. Visual Studio 2005 has tightened security to require that you specifically name the server to register. No, you won't be able to browse servers from this interface. Yes, you can browse for visible SQL Server instances from the ConnectionString dialogs, as I discuss in Chapter 9.

To register a server so that it appears in the Server Explorer Servers list, you'll need to right-click on the Servers icon, choose "Add Server...", and fill in the name of the server to explore. Sure, you can also specify that you want to connect to the server using alternative credentials.

Once a server is registered, an exploded view of the Servers icon lists the available root elements to explore. You can drill into each icon for additional elements, services, and logs to explore. I've expanded the Application Event logs in Figure 4.82 to give you an idea of what I'm talking about.

Figure 4.82. The Servers icon can expose logs, objects, and services to manage.

The Server Explorer "Servers" icon can expose:

  • Crystal Report Services: Assuming you have installed the BusinessObjects Enterprise .NET SDK from a Crystal Reports Server or BusinessObjects Enterprise, you can manage services associated with Crystal reports from this icon.

  • Event Logs: This icon exposes (at least) the Application, Security, and System event logs, as well as any application-defined logs. Keep in mind that each instance of SQL Server creates and uses its own Event Log, so this is a useful place to explore when you're trying to diagnose issues with a specific SQL Server instance. Let's take a closer look at these logs. When I click on the "+" next to the Application event logs and click the "+" next to the MSSQL$SS2K5 icon, I can see the copious list of events generated by the SQL Server 2005 instance running on the selected server.

  • Management Classes: The Server icon now permits developers to explore the .NET Framework's System.Management namespace via the Windows Management Instrumentation (WMI) infrastructure. The Management Classes icon exposes a long list of WMI classes to manage specific Windows operational metrics. I'll let you explore this icon, but consider that virtually every aspect of Windows is exposed here. For example, if you drill into Management Classes | Disk Volumes | C:, the properties page exposes important details like FreeSpace and the MediaType. Yes, some of these properties are RW, so they can be changed interactively in the Properties dialog. If you want to manage these classes in your application, Visual Studio can (in some cases) generate a custom managed class to do soright-click on one of the classes and choose "Generate Managed Class".

  • Management Events: This icon permits you to define events that fire based on specific WMI operations.

  • Message Queues: If Message Queuing has been installed on your system, you can use this icon to explore available message queues.

  • Performance Counters: The .NET Framework exposes a wealth of performance counters that can be used programmatically or from the Performance Monitor application to monitor the status of a wide variety of performance metrics. Unfortunately, the Server Explorer does not return the value of the performance countersit simply enumerates the counters visible at the time. Note that performance counters are created on-the-fly by applications as each instance starts. I expect to write an EBook on using and managing performance counters at a later date.

  • Services: There are a litany of services management interfaces provided by Visual Studio, SQL Server, and the Windows MMC Services console application. They all enumerate selected or all visible services installed on the targeted system. As with these other tools, the Services icon permits you to enumerate (all) visible servicesrunning or not. The icons listed under Services also show the current state of the service, as shown in Figure 4.83. Note that the SQL Server instances are shown with the instance name in parenthesis, as in SQL Server (SS2K5).

    Figure 4.83. Listing services under the "Services" icon in the Servers section of the Server Explorer.

To make managing the listed services easier, Visual Studio can generate a class to manage any selected serviceopen the Form designer, right-click on the service, and choose "Add to Designer" (as shown in Figure 4.84.) Note that this same dialog can be used to stop, start, pause, or continue some of the listed servicesand any of the SQL Server instances.

Figure 4.84. Using the Services icon to manage a selected service.

Warning

A bug in Visual Studio 2005 prevents managing a service if you add a service controller to the designer and subsequently remove it.

Once you click "Add to Designer", Visual Studio generates a class implementing a ServiceController instance to manage the service and places an icon for this class in the tool tray beneath the working Form. This class can be used to manage most aspects of the specified service. I'll discuss using the ServiceController class to manage SQL Server instances in Chapter 9.


Summary

This chapter introduced you to the Visual Studio IDEespecially the Server Explorer, which plays a vital role in how you, the developer, manage and interact with SQL Server objects. I showed you how to enumerate the contents of a SQL Server database (or any connected data source), as well as how to create new databases, tables, views, and other objects. The next chapter discusses how to use this same interface to work with stored procedures.


Chapter 5. Managing Executables with the Server Explorer

    Introduction

    Creating and Editing Stored Procedures

    Creating New Stored Procedures

    Executing T-SQL with Run Selection

    Debugging Stored Procedures

    Debugging Stored Procedures on Remote Instances

    Debugging Stored Procedures from Code

    Using the Server Explorer to Manage Functions

    Summary


Introduction

I'm not quite done discussing the Visual Studio IDE and the tools it exposes to help build your data-centric applications. This chapter continues by focusing more closely on the Server Explorer and how it can be used to manage T-SQL executables like stored procedures and functions. This includes coding, building parameter blocks, and saving and locating existing stored procedures and functions. I'll also discuss stored procedure-debugging techniques, including local and remote system debugging. Functions can be interactively debugged as well, so I'll discuss several techniques to test and step through your functions. As I'll illustrate in detail, all of this functionality can be managedcreated, explored, and debuggedwith the Server Explorer. Yes, much of this same functionality can be handled by the SQL Server Management Studio Object Explorer as wellin much the same way. While the IDE is a bit different, there are many similaritiesit's almost as if (almost) the two tools were created by the same company with the same goals. But it's late and I must be getting tired.

Since many of the designs I implement and see implemented all over the world are based on stored procedures, I'm glad to see the ability to manage stored procedures and functions woven so deeply into the fabric of Visual Studio. The Server Explorer does a pretty good job of managing these executablesit can locate and rename stored procedures and functions, as well as expose their parameters. Using the new "view by schema" option, you can also find pertinent executable objects more easily. If you have the Professional or Team editions, you'll also be able to step into (debug) stored procedures and functions from the Server Explorer. In any edition, you can edit these executables and even get the Query Designer to help generate SQL for them.

You don't need to create a Visual Basic .NET or C# application to use the Server Explorer to manage stored procedures, functions, or build queries. However, if you want to save the queries you're working on, you'll want to create a "Database" project.


Creating and Editing Stored Procedures

A pesky problem with the Server Explorer's stored procedure-management interface is that it has not really evolved very much since the Visual Basic days. There still is no way to categorize procedures into logical groups, and it does not differentiate between the stored procedures you write or are installed by Visual Studio or other "systems." This means the list of stored procedures you're forced to work with can be a seemingly endless list that scrolls off the screenmany of which don't really apply to your application or your current focus. Yes, I've seen how some application databases have dealt with this shortcoming by renaming stored procedures to facilitate their management. For example, "Accounting_FindRows", "Inventory_FindRows", etc. procedure names make it easier to locate application-specific or function-specific procedures in the list of stored procedures visible in the Server Explorer, but you still have to scroll through these to find the right stored procedure. Before long, you'll find that not all of the stored procedures are "visible" to your login credentials. That's because I've been encouraging DBAs to hide everything and expose only specific objects to developerson a "need to know" basis only. If your login credentials do not have rights to an object, SQL Server hides it from you. This problem has been handled quite elegantly in SQL Server Management Studio by exposing a "Filter" on the stored procedure list so you can filter (in) just those stored procedures you're working with.

Comparing SQL Server 2005 and Visual Studio 2005

Again, the SQL Server and Visual Studio teams have fallen (well) short in their attempt to integrate their UIs, so the SQL Enterprise Studio's Object Explorer and Visual Studio Server Explorer work and look about the same. As it is, they use different (albeit similar) metaphors to organize and manage the objects in SQL Server databases. I think, over time, you'll agree with us that the SQL Server Object Explorer is better suited to locate and manage stored procedures. I expect that the Visual Studio UI has been genericized and "dumbed down" to provide a level of compatibility between the various "foreign" (third-party) data sources it has to support. Consider that the Visual Studio tools had to adapt to non-SQL Server DBMS engines like Oracle and JET, as well as the OSFA[1] interfaces like ODBC and OLE DB. Not being a fan of OSFA UI, I'm not thrilled with this approach. Check out the side-by-side comparison of the two UIs shown in Figure 5.1. It's a shame these teams could not have come to agree on a solution that better mapped the SQL Server objects.

[1] OSFA: One size fits all.

Figure 5.1. SQL Server 2005 Object Explorer and Visual Studio 2005 Server Explorer.

The serious problem (that I've mentioned before) is that the Server Explorer does not expose the Security tab, so you won't be able to use the Server Explorer to manage logins, users, schemas, or rights without switching to an entirely separate program. The Server Explorer does permit you to alter CHECK Constraints (via the database diagram), but it does not support Rules (which Microsoft no longer wants to support) or Defaults (which it does support). No, it just looks like they ran out of time and could not get all of it doneI guess that almost four years wasn't enough. (Or is it eight years since the Server Explorer was first introduced in Visual Studio?)

Enter the SQL Server Management Studio Express

The newest member of the SQL Server tool suite is SQL Server Management Studio Express. When Microsoft first announced its new tool for the SQL Server Express edition, they were panned by me and the rest of the pundit community. Generally, I wanted something that was virtually identical to the SQL Server Enterprise Studio toolset but gated to the Express featuresand worked. The prototype tools were pretty shabby. I was concerned that a developer would have to learn two entirely different sets of tools to do basically the same thing.

Basically, we (all) got what we asked forthe SQL Server Management Studio Express is a clone of the SQL Server Management Studio that ships with the other versions.


Creating New Stored Procedures

The process used to create and edit stored procedures has changed very little since Visual Studio 2003 (and not that much since Visual Basic 6.0), so it should be familiar to those of you that have worked with previous versions. For the rest of you, the process is intuitive and should not present any real problems.

Tip

The Professional and better versions of Visual Studio 2005 and the SQL Server Management Studio tools permit you to create stored procedures, execute them, and run selected T-SQL code from the stored procedure code editor window, as discussed next in this chapter.

Remember, a stored procedure is simply a program written in T-SQL that runs on SQL Server. It might accept input and return OUTPUT parameters and always returns a RETURN value. Input parameters are used as placeholders for constants in the stored procedure logic. They cannot be used to name tables, columns, or other database objects. When you test your stored procedure using the Server Explorer, Visual Studio prompts you to supply parameter values. Note that Visual Studio sometimes gets confused when using OUTPUT parametersit might prompt for input values for these parameters as well. The best thing to do is to provide some valuewhich is ignored by the stored procedure during execution.

1.
Let's walk through the process of creating a stored procedure using the Server Explorer. Open the Server Explorer and navigate to the Stored Procedures icon (as shown in Figure 5.1). Right-click and choose "Add New Stored Procedure". This opens the new stored procedure template, shown in Figure 5.2.

Figure 5.2. The Create New Stored Procedure template.

Note that when you first create a new stored procedure, you need to use the CREATE PROCEDURE T-SQL command. However, once it's saved, Visual Studio automatically changes the syntax to ALTER PROCEDURE because SQL Server won't let you create a stored procedure with the same namebut it will let you ALTER an existing procedure.

2.
The stored procedure name is prefixed with the owner of the schema, so you need to work with your DBA to determine the correct nameor simply leave it off, as the name defaults to "dbo". Name your new procedure so that it's easy to find and conforms to your team's naming conventions. No, don't begin your stored procedure name with "sp_" as some documentation suggeststhis prefix is reserved for systems stored procedures. If you use this prefix, SQL Server searches the master database for the procedure before searching your "user" database. I'm going to name my first new procedure "CustomerByState"[2]. It's designed to return a limited number of customer rows from a specific state.

[2] The sample Biblio database already has a "CustomerByState" stored procedure, so you might want to name your new stored procedure "CustomerByStateNew" or some such.

Don't prefix your stored procedure name with "sp_".

3.
Directly below the CREATE PROCEDURE command, you'll find a begin comment "/*" tag (Figure 5.2, line 2). This is placed in the template to comment out the parameters block. Since my procedure needs a few parameters (as most stored procedures do), remove the beginning "/*" (line 2) and ending "*/" (line 7) comment tags. Note that T-SQL comments are colored green so when you remove the comment tags, the text color changes to black and blue. Blue text terms are reserved words recognized by the T-SQL editor. As you edit the text in the T-SQL editor, Visual Studio color-codes comments, keywords, operators, and logic differently to help make it easier to know if the code is correctly entered. No, there is no intellisense when coding stored proceduresI've asked for this feature for decades without much progress.

4.
Since my stored procedure logic requires several parameters, I need to populate the parameters block. For each parameter, I need to:

  • Name the parameter. In ADO.NET 2.0, parameter names are especially importantthey are the only way you can address specific parameters in the Command Parameters collection. The parameter name should be descriptive and make it clear what the developer needs to pass to the stored procedure.

  • Assign a datatype. Each parameter must be assigned one of the SQL Server datatypes or a datatype defined by a CLR datatype (as discussed in Chapter 13 Managing SQL Server CLR Executables). If the datatype is variable-length, you need to provide the maximum acceptable length.

  • (Optionally) Set a default value. If your procedure is written to permit the calling routine to call the stored procedure without including the parameter, you'll have to set a default value here in the stored procedure definition. If the ADO.NET Command Parameters collection does not include a Parameter object for a parameter with a default value, SQL Server substitutes the default value. If there is no default value set, SQL Server throws an error back to ADO.NET.

  • (Optionally) Set whether the parameter can accept a NULL. If your procedure is written to accept a NULL for the parameter value, you need to mark the parameter as permitting NULL.

  • (Optionally) Set the parameter direction. By default, all parameters are considered to be input to the stored procedure. If you want to pass back a value in a parameter, you need to mark the parameter as OUTPUT or INPUTOUTPUT (if the parameter should require an input value).

The stored procedure I'm building needs two parametersone to input the state code and another to indicate how many rows should be returned. Both of these parameters have default values set, so if the stored procedure is executed without passing parameters, it runs anyway but returns customers from all states.

5.
I'm ready to insert the T-SQL logic for the procedure. Under the AS clause, define any local variables used in the stored procedure. In this case, I'll illustrate use of local variables by declaring @Rows as an integer, as shown in Figure 5.4.

6.
Next, right-click in the stored procedure editor window directly before the "RETURN" and choose "Insert SQL". This opens the (now familiar) Query Builder, as shown in Figure 5.3 (after the T-SQL code was generated).

Figure 5.3. The "Insert SQL" dialog opens the Query Builder.

[View full size image]

When you're happy with the generated SQL, click "Validate Query" to test the SQL or "Execute Query" to test the logic. Since the stored procedure has parameters, Visual Studio exposes a dialog to capture the parameters, but this also means that the Validate Query option won't work. For some reason, this same dialog is exposed even if your stored procedure has only OUTPUT parameters. That's because the SQL Server interfaces have not been able to tell the difference between INPUTOUTPUT and OUTPUT parameters for some time. Note that the TOP expression cannot accept a NULL value or <default> as passed by the Visual Studio dialog that captures the input parameters, so you'll have to supply a valid value (at least, the first time). This dialog remembers what was entered for each parameter and uses that as the default for subsequent executions.

7.
When you're ready to save the generated SQL back to the stored procedure, click "OK". At this point, the generated SQL is inserted into your new stored procedure at the cursor, as shown in Figure 5.4. Note again that the code surrounded with the black border is an "atomic" code block.

Figure 5.4. The Query Buildergenerated SQL is inserted into your new stored procedure.

Add a SET statement to the T-SQL to capture the number of rows returned by the SELECT. I pass this value back as the RETURN value. Sure, I could have passed the @@RowCount directly to RETURN, but I wanted to show how to use a local variable.

IMHO

For some reason, the Visual Studio T-SQL editor insists on removing unnecessary (or, at least, what it judges to be unnecessary) carriage-returns (CR) in the T-SQL text. The T-SQL compiler does not care about these, and I often insert extra CRs into the T-SQL to make it easier to read (I did so in Figure 5.4). I think it's rude that the editor makes these arbitrary and unnecessary edits.

At this point, your stored procedure should be ready to save. Here is where the "fun" part starts. Most of us are accustomed to interactive code development IDEs that make it clear where I've made mistakes in syntax or usage. The SQL editor in Visual Studio is not one of those developer-friendly tools. If there's something wrong, the IDE returns a very terse error dialogalmost akin to "Stuff happened". For example, if you remove the parenthesis around the "(@MaxRows)" expression and click the Save button (or File Save), you get back the dialog shown in Figure 5.5.

Figure 5.5. The Visual Studio T-SQL editor fails to return the error line number.

Actually, this is better than previous versions that simply reported that the stored procedure could not be saved. It turns out that SQL Server returns more information about the syntax error, but Visual Studio throws it away. The same syntax error generates the following (see Figure 5.6) message in SQL Server Enterprise Studio (or in SQLCMD). Knowing the problem is on line 8 of the procedure is very helpful.

Figure 5.6. The SQL Server Enterprise Studio error message for the same syntax error.

Again, I expect that this oversight is because Visual Studio has to be an OSFA interface and they could not figure out how to genericize the UI and keep the (really important) line number that caused the problem.

Assuming you can get Visual Studio to save your new stored procedure, you can begin testing it from your application or from here in the Visual Studio IDE.

Note that the Visual Studiogenerated code is surrounded with a black bordercontaining what Microsoft calls a "SQL block." Note that if you need to change a SQL block, you can right-click on a block and choose "Design SQL block". Your stored procedure might have several SQL "blocks" of generated codeeach of these can be edited separately. As I said earlier, the power of the Query Builder is fairly limited, so you'll eventually bypass the generated code process to simply build and edit the code yourself.

8.
Now that your stored procedure has been saved, it should appear in the list of stored procedures in the Server Explorer. Click on the stored procedure icon in the Server Explorer and navigate down to your new stored procedure. Click on the + sign to expand the parameters list, and click on the first (@StateWanted) parameter. The parameter properties are displayed in the Visual Studio properties dialog (see Figure 5.7). Note that while these properties are read-only, you can glean enough information from these properties to create an ADO.NET Parameter object to map this parameter when it comes time to execute the stored procedure.

Figure 5.7. Focus on a specific stored procedure parameter to view the properties.

[View full size image]

Creating and Editing Complex Stored Procedures

Now that you've created a simple stored procedure, it's time you tried something a bit more challenging. The stored procedure I'm about to build is used to update a specific row in the Customers table and return the TimeStamp value from the changed row in an OUTPUT parameter. Yes, that's rightstored procedures are often used to perform complex DML commands.

1.
Navigate to the stored procedures icon in the Server Explorer and right-click. Choose "Add New Stored Procedure".

2.
Name your stored procedure "CustomerUpdate1". Yes, there is a "CustomerUpdate" stored procedure in the Biblio database that you can use as a finished solution.

3.
Set up a parameter signature to capture:

  • CustomerID as an integer

  • TID as a varchar(10)

  • CustomerName as a varchar(50)

  • Discount as a float

  • OriginalTimeStamp as a Timestamp

  • NewTimeStamp as a Timestamp

    Sure, you can name these parameters anything that makes sense for you and your team. The datatypes and lengths specified must match those used in the table definition. To determine the table schema, use the Server Explorer to navigate to the selected (Customers) table and drill into selected columns. The Visual Studio properties dialog will show the defined datatype, as shown in Figure 5.8.

    Figure 5.8. Using the Server Explorer to determine database table column properties.

    [View full size image]

4.
The NewTimeStamp parameter is set by the stored procedure code, so it needs to be marked as OUTPUT. If your code includes a parameter that serves as both an input and output parameter, you must set it to INPUTOUTPUT.

5.
The stored procedure has two T-SQL code blocksone to perform the UPDATE and a second to retrieve the newly generated TimeStamp value. These two operations are bound in a single transaction. Note that the UPDATE uses the Customer ID (CustID) to locate the row to change and uses the current TimeStamp value (fetched by the client application when the row was initially readbefore changes were made to the row) for concurrency. That is, if the original TimeStamp value matches the current (database) TimeStamp value, I know the row has not changed since last fetched, so it's safe to update. If the TimeStamp column has changed, I know some other operation has changed the row since it was last fetched. In this case, the UPDATE changes no rows. I inform the calling application of this by passing back 0 in the RETURN value and NULL in the @NewTimeStamp OUTPUT parameter.

Notice how the OUTPUT parameter is setit can be treated as any local variable. For example, it can be changed (set) in a SELECT or with a SET statement, as I have done on line 21 in Figure 5.9.

Figure 5.9. The CustomerUpdate procedure definition.

Remember, you should not depend on being able to fetch OUTPUT or RETURN value parameters until any rowsets are returned from the query. In this case, this is not an issue because I'm not generating a rowset. No, the SELECT @GeneratedTimeStamp does not return any rowsit's simply used to set a local variable. This means the OUTPUT and RETURN value parameters should yield correct results as soon as the stored procedure is executed.

Wait a second. I was going to skip this, as I'm running out of time, but I can't go any further without showing you how to use the SQL Profiler to monitor what SQL Server is executing. This tool should be one of the first you pull out of your bag when you start testing stored procedures or any code generated by Visual Studio or ADO.NET. I've included a few pages in Appendix III, "Monitoring SQL Server," to show you how to set up a basic trace. I suggest you take a look before going on.

6.
I'm ready to test this stored procedure, so navigate to the CustomerUpdate stored procedure in the Server Explorer and right-click. Choose "Execute". Since the stored procedure has parameters, Visual Studio displays the "Run Stored Procedure" dialog to capture the parameters (as shown in Figure 5.10). Note that all of the parameter are initially set to "<DEFAULT>".

Figure 5.10. The Visual Studio parameter capture dialog on first invocation.

Unfortunately, unless you manually set these values, Visual Studio executes the stored procedure using "default" for all of the parametersregardless of the datatype. This mean you'll have to fill in valid values (that conform to the datatype) or NULL for each parameter. If you want to pass NULL, use the dropdown list on each value to set "<NULL>".

The results of this execution are displayed in the Visual Studio Output window (View | Output). Use the dropdown in this window (if necessary) to view the Database Output (as shown in Figure 5.11). This includes a prototype for the execution (the actual T-SQL sent to SQL Server can be viewed in the profiler), as well as the built-in "rows affected" value and the values returned by any OUTPUT parameters and the RETURN value. Yes, to get (far) more detail, use the SQL Profiler. See Appendix III on setting up a SQL Profiler trace.

Figure 5.11. The Visual Studio Database Output window.

[View full size image]


Executing T-SQL with Run Selection

One of the features of the T-SQL stored procedure editor window that you might find especially helpful is the ability to execute arbitrary T-SQL code. As shown in Figure 5.12, you can create a "new stored procedure" to open a blank query window and enter whatever T-SQL you need to execute. Next, select one or more lines to execute, right-click, and choose "Run Selection". This executes the code (or tries to) and writes the output to the Database Output window (shown in Figure 5.11). This also means you can paste in or populate the T-SQL stored procedure editor from a batch file and execute the T-SQLincluding the GO separators.

Figure 5.12. Run Selection executes code from the stored procedure code editor.

[View full size image]

Tip

Run Selection is available in Standard Edition or better.


Debugging Stored Procedures

One of the most powerful tools exposed by the Server Explorer is the ability to step through an executing stored procedure. I call this T-SQL debugging. This technology has been around in Visual Basic and Visual Studio for some time now, but with a very checkered career. As the story goes, Drew Fletcher (a Visual Basic program manager) was walking near Times Square in New York when a man tried to get his attention from the shadows of a dark alley. "Hey, buddy!" he whispered in a gravely voice. "Youse want a way to debug stored procedures from Visual Basic?" Drew kept walking. He had heard about the nuts that hang out in the alleys off of 42nd Street and didn't need any extra adventure in his life. "Don't believe me?" the man persisted as he tailed Drew down the street close behind. "Beat it!" Drew half-shouted as he glanced over his shoulder. Fortunately, he recognized the stranger at oncehe was a Microsoft Consulting Services engineer that worked in the local office. To make a long story short, the MCS people in New York had been able to create a working prototype of a Visual Basic application that could actually debug and step through a stored procedure. What they didn't tell Drew is that this magic required DDE[3] to work. Over the years, the protocol changed as often as some engineers change their socks (or more frequently). That was the big problemgetting SQL Server to permit an attached development system to interact with the execution of the stored procedure without killing the system or other production work in progress.

[3] DDE: A low-level (and highly unstable) interop protocol used by Windows from the earliest days.

Today remote debugging has been implemented with yet another protocol. Hopefully, it works better than previous versions. Given that it has taken several wasted hours before finally being able to get it to work, I'm not so sure they have made much progress. At this point, some developers wish that Drew had kept walking and visited that guy up 42nd Street with the coat lined with "authentic" Rolex watches.

T-SQL Debugging Against a Local Instance

When T-SQL debugging works, being able to set breakpoints in stored procedures can dramatically improve your chances of figuring out what your stored procedure is and is not doing. The problem with this technology is that it can be especially hard to set up unless you're running Visual Studio 2005 on the same system as the SQL Server executing the stored procedure. To make things as clear (and safe) as possible, I'm going to try to lead you through this snake- and gator-infested swamp, so stay close together. Incidentally, T-SQL debugging is available only on the Professional or Team editions, so if you're working with the Express or Standard editions, you can only press your nose to the glass and watch.

Tip

Debugging Stored Procedures is available only on Professional and Team editions.

As I said, if you're trying to debug a stored procedure running on a local SQL Server (an instance running on the same system as Visual Studio 2005), it's easy (easier) to debug your stored procedure. Let's step through the process:

1.
From the Server Explorer, select the stored procedure to debug. In this example, I'm debugging "AuthorByYearBorn"it's one of the stored procedures included in the sample Biblio database (see Figure 5.13). Right-click on the stored procedure and choose "Step into Stored Procedure". This opens the now familiar "Run Stored Procedure" parameter capture dialog, shown in Figure 5.13.

Figure 5.13. Running a stored procedure with parameters displays the parameter capture dialog.

2.
When you click "OK", Visual Studio opens an interactive T-SQL debug window. Note the debug cursor stops at the top of the first block of executable T-SQL code (see Figure 5.14). Each block delineates a single statement of T-SQL codeeven though it might span several lines. Visual Studio highlights the block about to be executed (as with normal breakpoint debugging). The debugger steps through the procedure block by block (statement by statement), as expected.

Figure 5.14. Interactively stepping through T-SQL code.

[View full size image]

Once execution stops, you can hover over any of the T-SQL declared variables or parameters to see (or change) their current value. You can also hover over SQL Server-managed values (such as @@rowcount), but these can't be changed. As shown in Figure 5.15, I'm hovering over the @YearLow variable and Visual Studio reported its value (1940the value passed from the input parameter dialog). I then clicked on the value shown and changed it to 1939. As I step forward in the logic, this new value is used.

Figure 5.15. Any local variable (including input or output parameters) can be altered.

[View full size image]

Note that if the value set in an input parameter is NULL, Visual Studio shows an ellipsis "{...}" as the parameter value.

No, you can't change the T-SQL code (and continue execution), and you can't move the current execution pointer in a T-SQL debug session. Even with these limitations, it's still a useful toolat least, for simple SPs.

3.
To continue stepping through the code, press the keyboard shortcut assigned to Debug.StepInto. This instructs Visual Studio to step to the next code block. In Visual Basic, this key is F8 or F11. In C#, this key is F11.

4.
To continue the stored procedure execution to the end, press Debug.Start (F5) or click the green arrow in the Debug toolbar. Yes, T-SQL debugging works pretty much like a typical Visual Studio code-debugging session.

Debugging with Breakpoints Within Stored Procedures

Before you start debugging more complex stored procedures (those with more than a couple blocks of code), you're going to want to set one or more breakpoints. Just as when setting breakpoints in Visual Basic .NET or C# code, Visual Studio stops execution at the top line of the stored procedure code block containing the breakpointassuming you're not executing the application with the "Execute" option as previously described.

To get execution to stop on a breakpoint, you'll need to execute the stored procedure using ADO.NET in debug mode or interactively in Visual Studio using the "Step Into Stored Procedure" option. While Visual Studio 2005 has several new features that make it easier to debug stored procedures, it's still missing some important functionality, but I'll talk about that a bit later.

Setting a breakpoint is as easy as setting code breakpoints in Visual Studiosimply click on the left (gray) margin next to the point where execution should be paused. Breakpoints are indicated by a red ball in the margin, as shown in Figure 5.16.

Figure 5.16. Setting a breakpoint in a stored procedure.

[View full size image]

If you set a breakpoint in a stored procedure, the breakpoints are persisted with the Visual Studio IDE, so when that stored procedure is reopened, those breakpoints are restored.

For this exercise, I'm going to set T-SQL breakpoints in a stored procedure from the Biblio database (ComputeAvgPriceByPublisherAndType), which computes average book price by publisher and state.

1. | Using the Server Explorer, open a connection to the Biblio database, drill into the target stored procedures in the Biblio database, choose ComputeAvgPriceByPublisherAndType, right-click, and choose "Open".
2. | Choose the point to set the breakpoint and left-click in the gray area to the left of the line numbers. This sets a "standard" breakpoint as shown in Figure 5.16. Note that each defined code block (T-SQL statement) supports one (and only one) breakpoint. In my example, while I set the breakpoint at the COMPUTE statement (line 16), Visual Studio moves this breakpoint to the top of the code block (line 8). Additional breakpoints in the same code block are ignored. Once the breakpoint is set, Visual Studio and SQL Server interact to stop execution when the logic reaches the top of the code block (beginning of the T-SQL statement) when executing the stored procedure. Yes, this means if you execute this stored procedure from an application (in debug mode), Visual Studio reopens this window and permits you to step through the stored procedure as it executes. I'll show you how to do that later in this chapter.
Setting Multiple Breakpoints

Let's take a look at another aspect of step-through debugging. Set additional breakpoints on the stored procedure as shown in Figure 5.17.

Figure 5.17. The CompteAvgPriceByPublisherAndType stored procedure after a few additions.

In this example, I set breakpoints on lines 9, 21, and 23. Now when I choose "Step Into Stored Procedure", code execution pauses at line 9. When I press F5, execution pauses again at line 21. While paused, I can hover over any of the input parameters or local variables (such as @CustomerCount) to see the current values.

1.
When paused on the line 9 breakpoint, hover over the "@CompanyWanted" parameter. This opens an intellisense popup that includes the value, as shown in Figure 5.18. By clicking on the magnifying glass icon, you can choose which "visualizer" is to be used to display the data. If the data is complex, these can be a handy way to inspect the value. In this case, I set the @CompanyWanted parameter to "Beta".

Figure 5.18. Hovering the mouse cursor over a variable exposes the current value.

[View full size image]

2.
Click on the value ("Beta", in this case) and edit the value to change the current value of the selected variable. Moving the mouse cursor away from the variable closes the intellisense popup.

3.
Press F5 to step to the next breakpoint. Hover over the other variables to inspect the current values. Note that global variables (those beginning with "@@") cannot be changedthey're generated by SQL Server.

At this point, I've shown you how to step through a T-SQL stored procedure and set breakpoints on stored procedures running on a local instance of SQL Server. I'm ready to show you how to do the same from a "remote" (non-local) SQL Server instance.


Debugging Stored Procedures on Remote Instances

I spent quite some time trying to get T-SQL debugging to (finally) work against a remote SQL Server instance. I was initially distracted by the MSDN documentation that shows how to set up "remote" debugging. It turns out the solution was really pretty easy, but it's like saying it's easy to get into the Pentagon if you're the Secretary of Defense.

As a "best practice," I create a special local system account specifically dedicated to SQL Server and the related services it starts. This account is designated when SQL Server is initially installed. However, if SQL Server is running under this account, you won't be able to debug stored procedures remotelyunless you execute the Visual Studio IDE "as" this or an identical account. If you set up SQL Server to log on as the same account you use to develop, you'll be able to debug stored procedures just fine without having to attach processes or start "remote" debugging.

To set up the Visual Studio 2005 session that can perform T-SQL debugging against a remote procedure, you need to make sure that:

  • You have a stable connection to the server. I suggest using the Server Explorer to open a connection to the target server and drill down to the target stored procedure to be debugged. You'll want to do this anyway to make sure the needed breakpoints are set.

  • You need to ensure that the account you're using to run Visual Studio and the account used to connect to SQL Server are members of the sysadmin role (group) on the SQL Server hosting the stored procedure.

  • The account used to connect to SQL Server (from your ADO.NET code) can be either your Windows user account (if you are using Windows authentication) or a SQL Server account that specifies appropriate user ID and password. When using Windows Authentication, this account may be a local system account or a Windows domain account. In either case, it must match the account used to run the SQL Server service.

Let's walk through a configuration that works. It might save you some time when you're on a deadline and try to get T-SQL debugging to work. You might need to cooperate with your DBA to configure the server, but then again, you might just need to determine the Windows account used to run the SQL Server service.

1.
Starting with the target SQL Server, open the SQL Server Configuration Manager. While addressing the target SQL Server machine, drill into the SQL Server 2005 Services, as shown in Figure 5.19. Note the name of the service running the instance of SQL Server hosting the stored procedure to be debugged.

Figure 5.19. Using the SQL Server Configuration Manager to inspect the SQL Server instance service.

[View full size image]

2.
Note that the SQL Server instance (SS2K5) is logged on as ".\SQLE". This means it's logged on as a local machine account (not a Windows domain account). Your instance of Visual Studio 2005 will need to run as this account, or you'll need to change the account used by the SQL Server instance to match what you use to run Visual Studio.

I need to make sure this account (SQLE, in this case) has sufficient rights to debug stored procedures. That's the next step.

3.
Using SQL Server Management Studio, open a connection to the target server and drill into the Server Roles tab (as shown in Figure 5.20).

Figure 5.20. Drilling down to the Server Roles using SQL Server Management Studio.

4.
Click on "sysadmin" and inspect the list of role members. Make sure the account used to run the SQL Server instance is listed (as shown in Figure 5.21). In this case, the BETAV9\SQLE account is listed as a member of the sysadmins role, so I can use this account to start Visual Studio 2005 when debugging stored procedures.

Figure 5.21. Listing the members of the sysadmin role.

[View full size image]

5.
Next, I need to start Visual Studio 2005 using the same credentials used to start the target instance of SQL Server hosting the stored procedure. This might be a bit tricky if you don't know the password of the account, so you still might need to actually speak to your DBA. Given a few days, you should be able to crack this passwordunless the DBA knows how to create and use strong passwords. In this case, it would be easier to take the DBA out to a nice lunch and beg for the password. Of course, if you're also the DBA, you should take yourself out to lunch.

Starting Visual Studio (or any application) using alternative Windows credentials is not that hardyou simply have to use the "run as" feature of the Windows desktop. Let's step through the process of setting up a separate shortcut for Visual Studio to run as using different credentials.

6.
Right-click on the Windows desktop and choose "New shortcut".

7.
Enter the path to the executable application to start or browse to its location. Since I want to start the Visual Studio 2005 development IDE, I used "C:\Program Files\Microsoft Visual Studio 8\Common7\IDE\devenv.exe".

8.
Click Next and enter the name for the shortcut. I used "Visual Studio 2005Run as". This creates the new shortcut on your desktop.

9.
I'm not done yet unless you want to enter the logon credentials each time you click the new Visual Studio "run as" shortcut. Right-click the new shortcut and choose "Properties", as shown in Figure 5.22. Change the Target to include "runas.exe /user:SQLE /savecred". Yes, you should change the "SQLE" username to match the name used to start the target SQL Server.

Figure 5.22. Creating a custom shortcut to launch Visual Studio using alternative credentials.

[View full size image]

The "savecred" option is available only on Windows XP Professional and Windows 2003 Server. For Windows XP Home, you'll have to enter the credentials manually.

10.
I also changed the comment and chose an icon for the shortcut (the runas defaults to a simple icon). To use the Visual Studio 2005 icon, navigate to the executable and choose one of the icons from the list exposed. You can drag the new shortcut to the quick launch bar, if desired.

11.
Test your icon by clicking on the new shortcut. Yes, you'll be prompted for the credentials, but only the first time you use it. After that, Visual Studio is launched with the alternative credentials.


Debugging Stored Procedures from Code

You can debug stored procedures interactively (as I've been discussing) or from your applications. There aren't many mysteries here once you're able to debug stored procedures on non-local SQL Server instances. Here are a few guidelines:

  • Again, make sure that you can open a connection to the instance of SQL Server hosting the stored procedure. This might be a local or remote instance. I typically use the Server Explorer and navigate to the stored procedure and check to see if the required breakpoints are set.

  • Make sure you're executing Visual Studio with the same credentials as used to log on the target SQL Server instance. See the previous discussion in this chapter for details.

  • Make sure the application settings enable SQL Server debugging. This setting is available from the "Settings" dialogright-click on the project in the Solution Explorer and choose "Properties". Click on the "Debug" tab and check "Enable SQL Server Debugging", as shown in Figure 5.23.

    Figure 5.23. Enable SQL Server debugging in your project settings.

    [View full size image]

  • Make sure that the Visual Studio IDE is set to "debug" and not "release" mode. This is easy to set or determine when you use the "General Development" or "C# Developer" IDE settingsjust check the Solution Configuration toolbar drop down. However, it's not so easy when using the "Visual Basic Developer" IDE settings, as I discussed early in the chapter. If you don't see the Solution Configurations menu on your toolbar, you'll need to enable it. T-SQL debugging is disabled (for obvious reasons) in the "release" configuration.

Once you're ready, start your application as usualof course, it should include code to call the stored procedure to be debugged. When the logic reaches the line of T-SQL code containing the breakpoint, the Visual Studio IDE opens the stored procedure window and positions to the line about to be executed.

Note that the stored procedure to be debugged might not be the stored procedure called by your applicationit might be a stored procedure called by some other stored procedure or trigger that your application invokes directly or indirectly. For example, if the stored procedure to be debugged is called by (or is) a trigger, SQL Server might not execute the target stored procedure until a row is added, deleted, or updated (firing the trigger).

No, in most cases, Visual Studio won't complain about not being able to access a remote instance of SQL Serverit simply does not show the stored procedure window.

A word of caution: When debugging stored procedures, a considerable amount of TDS traffic is generated along with an equally heavy load on the target SQL Server. Do not enable T-SQL debugging if you don't plan to use it.


Using the Server Explorer to Manage Functions

Before I leave this chapter, let's take a look at T-SQL Functionsthey are similar to stored procedures in a number of ways. I've had the ability to manage SQL Server Functions since the earliest versions of Visual Studioas long as you were accessing SQL Server 2000 or later. A SQL Server Function (like its Visual Basic counterpart) is simply a block of code used to perform a set of logic and return a value based on zero or (almost) any number of input parameters. A Function, like a VIEW, can return a rowset, but unlike a VIEW, a Function can execute several SELECTs and other T-SQL or CLR logic as well. A VIEW can JOIN to the TABLE returned by a Function or simply return a Function-modified column.

T-SQL (user-defined) Functions are built for the traditional reasons:

  • Functions help us build modular stored procedures, queries, and VIEWs by encapsulating reusable logic. I also use them to simplify building code-intensive routines. This improves developer productivity.

  • Functions can be used in a T-SQL SELECT, WHERE, or CASE statement or to create a JOIN. They're also easier to invoke than stored procedures from within another SQL statement.

  • Compiled query plans used by Functions are cached, thus improving overall application performance.

Unlike a stored procedure, a T-SQL Function can't be used to execute DML statementsthey can't be used to modify data.

As I discuss later in Chapter 13, Functions can be written in T-SQL or in any of the CLR languages (like Visual Basic .NET or C#). I'm going to stick with T-SQL Functions in this chapter and show how to create, locate, edit, and debug Functions using the Server Explorer. SQL Server supports a wealth of pre-written Functions. There are Functions to return rowsets, Functions to perform complex aggregation or ranking operations, as well as configuration, server-side cursor, data-time, mathematical, and many other Functions you'll find useful, if not essential. I suggest you consult BOL for a list of all supported Functionsbefore you start building a Function that might already be included in SQL Server.

The value returned by the Function defines the type of user-defined Function (UDF):

  • A scalar UDF returns a single value.

  • An Inline Function returns a single TABLE type variable created by a SELECT statement against an existing table or VIEW. Inline Functions are generally fairly simple and are similar in functionality to a T-SQL View. Think of the Inline Function as a VIEW with parameters.

  • A "multi-statement" (or Table-type) UDF table Function returns a variable typed as a T-SQL TABLE object. Its structure is defined in the Function (using an INSERT INTO statement). Generally, this type of Function is most like a stored procedure in that the logic is more complex.

Unlike a stored procedure, a Function can return only a single TABLE-type variable, while a stored procedure can return any number of rowsets. Remember, while Functions are designed to be called from within stored procedures or other Functions, they can be called directly from your application. They cannot call stored procedures or execute "dynamic" (created on-the-fly) SQL. A Function also cannot call certain "non-deterministic" built-in Functions (those that don't return the same value each time, like RAND) or extended stored procedures[4]. Error handling is also very limited, as a Function cannot reference RAISERROR or @@ERROR. No, you won't be able to set breakpoints in a Function, but you can "step through" an existing Function to test it.

[4] See "Deterministic and Nondeterministic Functions" in SQL Server BOL.

Tip

No, you won't be able to set breakpoints in a Function, but you can "step through" an existing Function to test it.

Let's take a closer look at the structure of a Function. While similar to a stored procedure, there are some important differencesespecially in regard to defining how the data is to be returned. Like stored procedures, user-defined Functions (UDF) include:

  • A name that includes the owner. This means a UDF must be invoked by name and includes the owner name as: DBO.MyFunction.

  • One or more input parameter definitions, including options for each parameter. Note that T-SQL Functions can't return output parameters[5]. Parameters serve as constant placeholders whose value is set at runtime. It cannot take the place of a table n0ame, column name, or other database object. Each Function parameter can be assigned a default value. When a parameter of your Function has a default value assigned, you must call the Function with the keyword DEFAULT to use the default valueunlike the way stored procedures are executed. You can't just leave off the parameter.

    [5] CLR functions can designate specific parameters to be returned to the calling routine.

  • For Scalar and Multi-statement Functions, a BEGIN statement that frames the executable T-SQL code and an executable T-SQL body that perform the Function's logic. T-SQL Functions can (now) be programmed to call out to CLR logic. I'll discuss that and many other aspects of CLR functionality in Chapter 13. Note the limitations I mentioned earlier regarding use of non-deterministic Function calls, use of DML, and error-handling/reporting Functions. The T-SQL statement body is ended with an END statement that marks the end of the executable T-SQL code.

  • A RETURN parameter definition and options. Functions can return a scalar (a single value), a rowset from a single SELECT, or a rowset in the form of a TABLE object. The RETURN type can be any type (except Timestamp), including CLR user-defined types.

Creating T-SQL Functions with the Server Explorer

I'm ready to step through the process of creating three new T-SQL Functions using the Server Explorer. In the following example, I start by illustrating the code to create a simple Inline Function that compares two strings to see if they sound alike. This T-SQL uses the built-in T-SQL SOUNDEX Function to do most of the work.

Tip

Sure, this could have been done by simply coding an expression that called the built-in SOUNDEX Functions directly.

1.
To get started, as with stored procedures, drill into an existing Data Connection in the Server Explorer and navigate to the Functions tab.

2.
Right-click on the Functions tab and choose "Add New". This opens a selection menu (as shown in Figure 5.24) that permits you to choose the Function template. There is a template for each type of FunctionI'll create one of each in the following exercises.

Figure 5.24. Creating a new Function with the Server Explorer.

3.
Choose "Scalar-valued Function". This opens a T-SQL editor window with the Scalar Function template read to be completed with your code (as shown in Figure 5.25).

Figure 5.25. The Scalar Function template.

Note how the RETURN statement datatype is left out. This needs to be set to correspond to the datatype to be returned by the Function. Yes, you can use any type (except for timestamp). The logic for the Scalar Function is inserted between the BEGIN and RETURN (the RETURN must be the last statement in the code). This is called the "Function body"it's specified only in Scalar and Multi-statement type Functions.

4.
Add parameters and code to your new Scalar Function. In this case, I need to capture three parameters and return a string (varchar). The parameters input two strings to compare and a "difference" criterion. The DIFFERENCE Function compares the two strings and returns a value (04) to indicate how closely the two strings sound alike. This approach is useful when searching the database for a name that "sounds like" some other word. The completed Scalar Function is shown in Figure 5.26.

Figure 5.26. The completed Scalar Function.

[View full size image]

5.
To save the Function, click the Save icon or click the File | Save <Function name> menu. If you've made a syntax error, Visual Studio returns a simple (and relatively useless) error message. If you get stuck, try pasting the code into a SQL Server Management Studio query windowthe syntax exception returned by this editor includes the (very useful) line number of the incorrect line of code.

You can test this Function using the Server Explorer or by using another T-SQL script, as I've discussed earlier in this chapter. I'll step through some testing scenarios after I discuss the next type of Function.

Creating an Inline Function

Inline Functions are less complicatedespecially when the Server Explorer template helps get you started. In this case, the Inline Function is programmed to return a T-SQL TABLE object built from a single SELECT. Sure, this SELECT can JOIN two or more tables; include an ORDER BY, GROUP BY; or reference a VIEW or other Functionsjust as long as it returns a single resultset containing a rowset. Let's step through the process of creating the Inline Function.

1.
Choose "Inline Function". This opens a T-SQL editor window with the Inline Function template ready to be filled in (as shown in Figure 5.27).

Figure 5.27. The Inline Function template.

2.
Note that the template is fairly simple. It defaults to returning a TABLE-type variable generated from a single SELECT. Remember to think of the Inline Function as a VIEW with Parameters. For this example, I'm going to create a Function that returns a TABLE containing customer IDs (CustID) from customers living in a selected state. Once the SELECT is generated, it appears imbedded in the query (as shown in Figure 5.28).

Figure 5.28. A simple user-defined T-SQL Inline Function.

3.
To save the Function, click the Save icon or click the File | Save <Function name> menu. If you've made a syntax error, Visual Studio returns a simple (and relatively useless) error message. If you get stuck, try pasting the code into a SQL Server Management Studio query windowthe syntax exception returned by this editor includes the (very useful) line number of the incorrect line of code.

4.
At this point, I'm ready to test the Function. Since the Inline Function returns a TABLE object, you can address the columns using normal syntax. A simple test executes a SELECT against the Function-generated TABLE. To set this up, open a new stored procedure window (right-click on the Stored Procedures tab and choose "New Stored Procedure"). Next, enter the test T-SQL (as shown in Figure 5.29). Yes, I'm using SELECT * here to test the Functionto ensure that all of the required columns are returned.

Figure 5.29. Testing the new Inline Function in a stored procedure code edit window.

5.
Once the code is ready to execute, select one or both lines of the code in the stored procedure code edit window and right-clickchoose "Run selection" to test the Function. The results are shown in the Visual Studio Output (Database Output) window (at the bottom of the IDE), as shown in Figure 5.30.

Figure 5.30. Executing code with Run Selection from the stored procedure code edit window.

Creating a Table Function

The Table Function is a bit more complex, but again, the Visual Studio Function template can give us a good running start at creating it. This type of Function is called a "Table" Function in Visual Studio but is referred to as a "Multi-Statement" Function in the SQL Server documentation and the earlier discussion in this chapter. It's different from the Inline Function in that it includes code to generate the TABLE it returns in the body of the logic. No, the "signature" of the function can't change from execution to execution, so the Function always returns the same TABLE object columns.

In this case, I'm going to create a particularly useful Table Functionit converts a delimited string into a TABLE object. While that might not seem like much, this Function can be very useful when building multi-choice queries. That is, if you're generating an application that permits the user to choose from a list of pick-list options, this function can convert the delimited string returned by the control to a TABLE object that can be used in a JOIN. I'll illustrate this in an example a bit later.

1.
Start by right-clicking the Functions tabchoose "Table-valued Function" from the "Add New" function. This opens a new Function code edit window, as shown in Figure 5.31.

Figure 5.31. The Table-type Function template.

[View full size image]

Note that the RETURNS statement defines the signature of the TABLE object created by the function. It's exposed to the Function code block by the @table_variable, which is scoped within the Function. You're not permitted to execute any SQL function that returns values other than those returned by the TABLE variable in the RETURN statement. Let's step through the Table Function syntax so I'll better understand the code when I explore the example.

2.
Next, define the input parameters for the Function. Just as before, define each parameter's datatype and (optionally) a default value. Defaults are set so the Function can be executed without passing a value for the parameter (but you will have to use the keyword "DEFAULT" in place of the parameter).

3.
Next, set up the RETURNS statement by defining the TABLE object to be returned. This statement resembles a parameter blockand it is, in a sense. It defines each column of the new TABLE object I'll be returningsort of like "output" parameters, but in this case, I'll return many rows of data instead of a set of OUTPUT parameters, as I do with stored procedures. My simple TABLE object has only two columnsan integer row number and a string value. These first sections of the TABLE-type function are shown in Figure 5.32.

Figure 5.32. The parameter block and RETURNS statement for the Table-type Function.

4.
At this point, I'm ready to fill in the code block. I frame the code block with BEGIN and END statementsin between, I code the Function logic, finishing with the RETURN statement. The Function must end with a RETURN statement.

In this example, I use the (rather limited) T-SQL string functions to parse the delimited string passed to the Function. The delimiter is passed to the Function as a parameter, so the Function includes logic to validate the input parameters as well as the length of the string to parse. Since the Function can't raise an error or throw an exception, I don't have a lot of options to report invalid parameters at runtimeI return an empty TABLE if the arguments aren't within spec. The code for these validation tests are shown in Figure 5.33.

Figure 5.33. Validating the Function input parameters.

5.
The rest of the code block parses the delimited string passed to the function. It walks the string and uses the INSERT statement to add rows to the in-memory TABLE object I've defined. The rest of the code block is shown in Figure 5.34.

Figure 5.34. Parsing the delimited string and populating the TABLE object.

6.
To save the Function, click the Save icon or click the File | Save <Function name> menu. Once saved, the Function appears in the list of Functions exposed by the Server Explorer.

I'm ready to test the function, so (as promised) I'll create a query that uses the StripToTable Function to use a delimited list to drive a T-SQL IN statement. Again, I used the Query Designer and the empty stored procedure code editor window to create and test the query, as shown in Figure 5.35.

Figure 5.35. Testing the Table-type Function.

Note that the IN statement ordinarily accepts a delimited string, but this string must be fixed at compile time. By using the StripToTable Function, you can provide a delimited string as an input parameter to a stored procedure that duplicates the functionality shown in Figure 5.35. Let's take that extra step and create a stored procedure that accepts a delimited string and returns a customer list. Figure 5.36 illustrates this code.

Figure 5.36. Using the Table-type Function in a stored procedure.

I get the same results as when I tested Figure 5.35 by executing the following code (shown in Figure 5.37.)

Figure 5.37. Testing the stored procedure that executes the Table-type Function.

Previous Page
Next Page


Summary

This chapter shows how the Visual Studio 2005 Server Explorer can be used to create, locate, organize, and debug SQL Server stored procedures and functions. Along the way, I showed you how to use the "New stored procedure" window to execute ad-hoc queries or maintenance scripts to configure and test your server-side executables. Sure, I would like to see a window dedicated to ad-hoc query debugging. It would also be nice if the ability to script SQL Server objects (as supported in SQL Server Management Studio) was incorporated here in the Server Explorerbut I'm not holding my breath. Remember that this is an OSFA interface, and Microsoft feels that if a feature is made available for one DBMS, it must be made available for all. I think that's like saying that if a Kansas City butcher sells beef, they should also sell fresh salmonand soy burgers for the vegetarians. When you're all things to all people, you're nothing for everyone.


Chapter 6. Building Data Sources, DataSets, and TableAdapters

This chapter explores the new Visual Studio 2005 paradigm used to create and use "strongly typed" TableAdapter classes using the Data Source Configuration wizard. A strongly typed data structure (as I discuss in detail later in this chapter) is simply a class that exposes the columns of a rowset as property variables whose datatype has been assigned based on the metadata gleaned from the SELECT query (at design time). As you'll figure out before long, I'm not a whole-hearted fan of this approach, for a number of reasons that will become apparent as you read the chapter. Suffice it to say that a strongly typed data structure does not necessarily buy you much beyond the ability to fetch data from memory (a lot) faster and provide what's known as "type safety." That is, a type-safe data structure can't accept a value outside of the type. Integers can accept only properly sized integers, floats can take only property scaled floats, and strings can take only strings. That's cool, but that does not mean the data is validjust the right type, but I'm getting ahead of myself.

In earlier versions of Visual Studio, you could use the Generate DataSet wizard to construct strongly typed DataSet classes. I'll talk about that approach, as well, later in this chapterno, it's not where you might expect to find it. The entire paradigm is built around strongly typed classes. As you'll see, this concept is the hinge-pin of Microsoft's push toward its Service-Oriented Architecture (SOA). It continues to promise better performance, better developer productivity, better data purity, and more work for consultants who understand how to use it. Just how well it delivers on those promises is a matter of discussion.

In this chapter, I'll start by touring the process of designing and modifying Data Sources and TableAdapter classes, as well as creating strongly typed DataSet classes. You'll see where these classes make sense and make it clear where they just get in the way. Along the way, I'll briefly point out the XML structures used to implement these classes, but I don't plan to wade into that swamp. At least one of us does not think a developer needs to edit XML to get his or her job donebut it doesn't hurt to know what's going on behind the curtains. When I discuss the ReportViewer control in Chapter 14 "Creating and Managing Reports", you'll see how creating a Visual Studio Report can be facilitated by using the Data Source Configuration wizard (DSCW) and TableAdapter Configuration wizard (TACW) to auto-generate a TableAdapter, but you'll also see how that's really not necessary. I'll also show you how to actually use these generated code classes in your applications.

IMHO

Frankly, the Data Source and TableAdapter class make impressive, virtually "codeless" demos, but don't feel threatenedI don't think they're going to completely replace the connection management, data access, and binding code in serious applications.

Incidentally, you won't find the Data Source Configuration wizard when you open a Web Service or ASP.NET projectthey aren't supported in those development modes in Visual Studio 2005. For those project types, you're still going to have to use the Server Explorer to expose data sources (Tables, Views, and stored procedures) and drag and drop these objects to your ASP forms or bound controls.


Why Create YADAI?

The "DataWorks" team at Microsoft (the folks who invented ADO.NET) has been hard at work reinventing ADO.NET (again), and, yes, they managed to create yet another data access interface (YADAI) for Visual Studio 2005. To be fair, the DataWorks team feels the need to build data access and user interfaces to help developers from all levels of experience, from rank amateur to front-line professional. While the classes I discuss in this chapter won't help everyone, they can help some developers as they try to build many basic-to-intermediate applications. If you delve into SQL Server Everywhere you'll also find that these wizards and classes are an almost perfect fit. That's because the SSEv engine is designed to support small, single-user databases. Sure, developers (and applications) might outgrow these drag-and-drop and pick-from-a-dizzying-list-of-options UI code generators, but until they do, it's best to understand how they work, what they are capable of doing, and (most importantly) what they can't be expected to do. And, yes, these tools can help build serious applications, but with a number of limitations that I discuss throughout the book.

The DataWorks team has been struggling with a number of problems. Some of these include:

  • The DataAdapter Configuration wizard generates code that simply populates an untyped DataSet. Microsoft wants developers to create strongly typed DataSet and DataTable classes. They tell us that strongly typed data structures with their "type-safe" data elements are more secure and easier to work with.

  • The inability of the DataAdapter to support custom (developer-defined) Fill methodsespecially those that accept parameters. This makes it difficult to use the DataAdapter to return rowsets from other than the one (and only) SelectCommand SQL query.

  • The DataAdapter Configuration wizard did not provide enough flexibility for a number of scenarios. The most important of these issues is the inability to determine how collision detection is to be performed.

  • The Update method makes a round-trip for each row in the target DataTable that's added, changed, or deleted. Round-trips are expensive, and this approach degrades performance.

  • Developers wanted to be able to alter the code generated by the DataAdapter Configuration wizard (DACW) but were thwarted, as any customization made to the generated class code was overlaid if the DACW was reexecuted.

  • The inability of the DataTable to be serialized or support DataSet methods needed by some developers. Many developers have felt that the DataSet imposed too heavy a burden when a single DataTable would have sufficed.

  • In theory, Visual Studio 2005 permits developers to generate code sourced from XML Web Services, custom business objects, as well as traditional relational data sources. Any new data access interface must support all of these sources.

  • ASP.NET Web Services and business objects are a growing source of data, and the current tools don't adequately help build applications to expose their data.

  • The new ReportViewer and other bound controls needed an easier way to define a Connection, queries, and rowsets needed to populate the reports and controls.

Do the Data Source Configuration wizard and TableAdapter address these problems? Yes, they doat least, to a large extent. Many of the other issues are dealt with by fundamental changes in the .NET Framework and the CLR languages.

Consider that Visual Studio has evolved over the last 8 years (or so) from a COM-based linkage builder to a code generator. In the Visual Basic 6.0 days, developers were encouraged to drag and drop (just as they are today), but the IDE was charged with connecting user-written code to the properties of canned (black-box) COM objects like the Data Environment Designer. The VB6 Data Object wizard (DOW) was the first prototype of the DataAdapter Configuration wizard, which further evolved into the Data Source Configuration wizard and TableAdapter code generators. Today, Visual Studio 2005 is a sophisticated code generatorthe volume and complexity of the code it generates is startling (and often intimidating). Fortunately, Visual Studio 2005 does a passable job of keeping this auto-generated code, hidden, correct, and protected from changes you make to other parts of the project.


Is Strongly Typed Data Important?

Some would have you believe that the only way to build a data structure is to use one of the strongly typed class code generators exposed in Visual Studio. Ah, developers all over the planet have been creating mainstream, high-performance, and highly scalable applications without strongly typed data structures for decades. So what makes a strongly typed DataSet special? Well, instead of declaring an object as a System.Data.DataTable and populating the Columns collection with hand-coded or ADO.NET-generated schema, you either tediously hardcode the thousands of lines of code to build individual columns as properties using specific system types in your own custom "AuthorsDataTable" or you get one of the Visual Studio tools to do it for you. That's what this chapter is abouthow to get Visual Studio to create the new strongly typed TableAdapter class for you.

Behind the scenes, a strongly typed data structure has a matching schema file (.XSD) that defines the structurewhat it should contain and how it should be stored. It can also define relationships between the classes as well as other constraints, custom properties, and methods. This XSD file is used to generate the code used to access the targeted database element.

An untyped DataTable (or any object) has no such defined schema (outside of the database), so it's built with generic system "object" types that can hold anythingit's simply a transport vehicle. When you execute a query that returns a rowset, the ADO.NET DataAdapter Fill or DataTable Load methods construct a DataTable whose Columns collection matches the schema of the incoming rowset. ADO.NET can identify a primary key but not intertable relationships. While the DataType property is set to match the type of the rowset column, it's implemented as an objectnot as a strongly typed property.

Think of a strongly typed DataTable as an egg carton and an untyped DataTable as a basket designed to carry anything you happen to collect. Does this mean the strongly typed DataTable can store egg-size chunks of stuff you might find in a barnyard? Sure. A city slicker won't know they aren't eggs until he tries to crack one open in the kitchen. However, if you want to ensure you only put real (chicken?) eggs in your strongly typed egg carton, you'll have to add custom code to test each item as it arrives. Each block of custom criteria-testing code you add further restricts what the carton can contain. It also means that if you buy a duck, you won't necessarily be able to use your strongly typed cartons without going back and modifying every single application that uses the strongly typed class to accept the new type of egg. Thanks to partial classes, Visual Studio 2005 does not overlay the customizations you make when you rerun the wizards used to create strongly typed data classes, so it can make this process a bit easierbut just a bit.

So what benefit does a strongly typed DataTable (or DataSet) return for your investment in time and the compile-time required for hundreds, possibly tens of thousands, of lines of brittle code? Yes, if a data structure (like a DataTable) is defined explicitly, it cannot accept values that are out of type. That is, if you define a strongly typed DataTable (as the Data Source Configuration wizard does), the columns marked as Int16 won't accept a string or an Int32, but does that make your data pure? Not hardly. A strongly typed Int16 field that's supposed to hold the customer discount (which can be 0, 10, 20, or 30%) is not prevented from holding any Int16 value unless you further edit the generated code to impose this business rule. As you can see, this can be very expensive to implement and even more expensive to maintain.

As I see it, the problem is and always has been that when the underlying data schema or business rules change (as they invariably do), the strongly typed class you or Visual Studio create no longer match the data structure as defined in the database. In our opinion, this is the more serious and insidious problem. From the dawn of time, data structure inconsistency (what I used to call "data exceptions") has caused any number of crippling problems that hard-coded data structures do not address.

One of the most important vulnerabilities of the Data Source paradigm is that the generated code is directly dependent on the database schema (structure) used to generate the code. If the schema changes at design time, Visual Studio usually gets confused and displays what I call the "white window of death." You're lucky, thoughthere's a work-around for this problem. Close the Form's design window, right-click on the Form file in the Solution Explorer, and click Refresh. The Form window should rebuild itself.

If the schema changes post-deployment, you're pooched. Applications that use the DSCW-generated code are hard-coded to match database table and view definitions, store procedure and function signatures, and user credentials. If any of these change, even in subtle ways, your code will likely crash. At this point, your only alternative is to rebuild, recompile, retest and redeploy, and possibly look for another customer.


What Is a Data Source?

A "Data Source" (according to Visual Studio 2005's new[1] definition) is a means of generating one or more TableAdapter classes. The Data Source XSD designer (shown in Figure 6.1) and the Data Source Configuration wizard are designed to provide a way to generate visualize relational tables, stored procedures, views, or objects exposed via an XML Web Service or business object interface. These Data Source objects expose named DataSet and TableAdapter classes as icons that can be dragged from the Data Sources window to a Windows Forma process that instructs Visual Studio to generate the connection, linkage, binding, navigation, and action queries code needed to display, capture parameters, show position, navigate through a query's rowset, and update the data on demand.

[1] The term "Data Source" has been used for a variety of classes, objects, concepts, and other random stuff for decades.

Figure 6.1. A populated Data Source Designer window.

[View full size image]

The code generated by the Data Source Configuration wizard is saved to a "Designer" file in your solution that can be inspected (but I don't recommend that you change it) to see how the class is implemented.

During manual configuration (a required step, as far as I'm concerned) you should:

  • Refine the query being executed. For example, you can add a WHERE or TOP clause to limit the number of rows returned.

  • Define parameters to further focus the query.

  • Designate what code and UI elements are generated when you drag the icon to a Windows or ASP.NET Form.

  • Designate the code methods used to populate the underlying custom strongly typed DataTable objects.

  • Add TableAdapter classes to the Data Source and tune the relationships between those classes.

  • Define or refine the code used to update the target rowset.

  • Specify how each TableAdapter column is to be rendered in bound controls.

  • You can choose appropriate Windows forms controls to display the column value or write a custom control to display it.

Tip

The .XSS file created by Visual Studio is used to map the objects shown in the Data Sources Designer window.

The aforementioned wizards also deal very nicely with query parameters (if you specify them) and provide a clean navigation tool to step from row to row to implement simple control binding. I built a sample of this UI by creating a "codeless" interface to the AuthorsByYearBorn stored procedureincluding the query parameters (as shown in Figure 6.2).

Figure 6.2. Data Source Configuration wizardgenerated code populated this form.

Simple control binding connects data from a data source (DataTable single Row value) to a single-item control (like a TextBox). In contrast, "complex" data binding is where you point the DataSource property at a DataTable (or, better yet, a DataView) and let the control display all rows at once.

While the Data Source Configuration wizard works pretty well for simple (and small) database tables, it quickly loses steam when you try to leverage it to access large tables, Web Services, or business objectsthese aren't nearly as intuitive, and you'll end up doing a lot more work than you might feel is warranted.

For those transitioning from earlier versions of Visual Studio, you might recall that strongly typed DataSet classes were created by their own wizard that was launched from the Data menu. Well, Microsoft has chosen to hide that wizardthey want you to use their new shiny Data Source wizard. Thankfully, the functionality and wizard are still there; you just have to turn over a few rocks to find itif you still want to use it. ASP and Web Service developers won't have a choice. They'll still have to use this "ancient" method to construct their strongly typed classes. I'll walk through this process later in this chapter.

As I said, the Data Sources XSD designer is driven by the Data Source Configuration wizard, which creates strongly typed TableAdapter classes. It accomplishes this by capturing (or helping you construct) a ConnectionString that identifies the source of the data, the user credentials, and other options used to establish a link (a connection) to the data. The ConnectionString you build is persisted to the app.config file and can be manipulated by using the new "Settings" tab of the Project property page. The interactive DSCW explores the targeted data source and enumerates all visible[2] objects. For a relational database, this means you'll see a dialog listing all of the Tables, Views, and stored procedures your credentials (those used to launch Visual Studio) have a right to see. For a Web Service or business object, you should see the methods the service or object exposes.

[2] The object "explorer" shows only objects that you have rights to see. I discuss rights later in this chapter.

Tip

I walk through the process of building a Data Source step by step later in this chapter.

In the process of running the DSCW, the UI populates a tree-view of the available objects and permits you to choose which of these to access. Once you select one or more objects, the DSCW generates a strongly typed TableAdapter for each of the selected objects and a (giant) XSD file that describes the Data Source in XML. By default, a "Fill" and "GetTable" method are generated for each objectreturning all of the rows from the objects using expensive SELECT queries with no WHERE clauses. For example, creating a Data Source against the four-column Authors table inserts 853 lines of code into the dsAuthors.Designer.vb and generates a 115-line XSD file. It does not take a nuclear scientist to know that it makes sense to be judicious about the number of objects you choose to access. No, you don't have to (and probably shouldn't) return all columns from selected objects. When working with Tables and Views, the wizard permits you to specify which columns to fetch. It also makes sense to tune the auto-generated queries to accept parameters that limit the scope of the SQL. I'll show you just how this is accomplished as I walk you through the DSCW step by step later in this chapter.

Make no mistakethe Data Sources window is not a "Data View" window or Server Explorer. When you include a Data Source in your application, the compiled code you distribute and load into memory includes the thousands of lines of generated code created by the DSCWeven if your code never touches a single one of the defined TableAdapter objects. The size of the compiled DLLs can significantly impact the performance of your application. For ASP.NET web sites, a larger memory footprint directly impacts the number of users you'll be able to support (scalability).


What Is a TableAdapter?

Basically, a TableAdapter is a code-generated strongly typed datasource-agnostic interface over underlying SqlClient (or OracleClient, Odbc, or OleDb) namespace objects. The DSCW generates the TableAdapter from whole cloth based on a specified query rowset, so don't go looking for a TableAdapter class in the object browser. However, the TableAdapter class is coded to inherit from the System.Data.DataTable and System.Data. DataColumn classes (as shown in Figure 6.3). This means that the TableAdapter supports all of the methods and properties of the base classes (the DataTable and DataColumn among others).

Figure 6.3. Part of the class used to implement the "AuthorsDataTable" in a TableAdapter.

[View full size image]

To manage updates, the TableAdapter is generated using a SqlDataAdapter (or other DBMS-dependent DataAdapter), including hard-coded SelectCommand and (optionally) UpdateCommand, DeleteCommand, and InsertCommand definitions. Ultimately, the TableAdapter is designed to populate a single DataTable through custom Fill and GetTable methods.

Nope, the DSCW does not ask you if you want to use TimeStamp, uniqueidentifier or more intelligent collision-detection schemes, but it does know how to detect a Timestamp column in the SELECT query used as a pattern to generate the code. That is, the generated UPDATE statement tests for post-fetch changes by comparing the current Timestamp value with the Timestamp value initially fetched. This seems to make it a good idea to add a Timestamp column to any database Table that you expect Visual Studio to update using generated code. The generated TableAdapter UPDATE query for the Authors table that includes a Timestamp column is shown in Figure 6.4.

Figure 6.4. Generated UPDATE statement for a Table that includes a Timestamp column.

[View full size image]

Incorporating Timestamp concurrency checking is a big step forward. While the DataWorks folks wanted to include even more functionality (to be on par with ADO classic), these features never made it to the final buildsbut I'll talk about that and what you can do to correct these shortcomings a bit later. Note that the generated code is hardly easy for humans to readeven though the DataWorks folks stuck in an extra CR/LF sequence for some reason. I'll talk about managing ADO.NET updates (DML) tasks in Chapter 12 "Managing Updates".

As you'll see later in this chapter, once the DSCW does its thing (and generates all that code), you'll have an opportunity (a duty) to go back in and reconfigure the generated code via a GUI. I'll also talk about the next step later in this chapterdata binding. This is the code needed to connect the data returned by the TableAdapter to one or more bound controls. You'll see how simply dragging icons from the Data Source window to the desktop auto-generates bound controls, navigation controls (to step from row to row), and TextBox controls to capture the input parameters.

What do you get when you step though this process that you could not have achieved with about 50 lines of code that you wrote (or leveraged) yourself? Ah, not much. Yes, the TableAdapter is a fast way to fetch data by nature of its strongly typed foundation, but I expect you'll quickly outgrow it.


What's Missing in the TableAdapter?

Not everyone likes the new Visual Studio 2005 TableAdapter class. According to Sahil Malik[3] (to paraphrase a recent newsgroup exchange)...

[3] Sahil is the author of Pro ADO.NET 2.0 (Apress, 2005).

  • TableAdapter classes "suck eggs" with Systems.Transactions and don't have built-in transaction support.

  • They are "single-table" centric. That is (if I understand his point), the TableAdapter encourages developers to simply point to a single database table and build a direct interface to selected columns from that (entire) table. As you'll see later in this chapter, you have to take several additional steps to refine (define) the WHERE clause or otherwise limit/focus the rowset.

  • They are "impossible to make work with multiple tables in hierarchical scenarios (or) concurrent environments." It seems that Microsoft's intention was to eliminate the need to manually code the Relation objects when building a hierarchical data structure. They did that, but does it really make it easier to manage these hierarchies with ADO.NET? That remains to be seen.

  • They are "unwieldy to maintain as your db (database) structures changes." I agree.

He goes on to list several other points, but you get the idea. Part of the problem is that the TableAdapter and Data Source interface is new and not fully understood or appreciated. That said, just because it's new does not make it "improved" or suitable for anything. This problem is exacerbated by hiding the old tools, which just confuses customers and encourages (forces) them to use the new toolsregardless of how well they address the problem at hand. No, I think the jury is still out on the TableAdapterand they aren't wearing smiles.


Creating Database-Based Data Sources

The Data Source Configuration wizard is one of the first things you'll see demonstrated if you watch Microsoft show off the new data access features of Visual Studio 2005. At this point, I'm going to step though some of these "demo" scenarios and discuss some of the details they might have failed to mention as they dragged-and-dropped their way across the stage. Later on, I'll show some other more sophisticated ways to leverage the DSCW.

Tip

The "Data Design" toolbar exposes basic Data Source operations like Generate DataSet, Show Data Sources, Preview Data, and Add New Data Source.

Building a Single-Table TableAdapter

The goal of this first exercise is to create a single-layer (non-hierarchical) interface to a database Table located on SQL Server. To generate the behind-the-scenes code, the DSCW has to gather enough information to:

  • Identify the "source" of the data: This means identifying the machine name of the system hosting the SQL Server instance and the instance name.

  • Specify the credentials: Your application expects the application to use these credentials to gain access to this server. There is a lot to these two stepsso much that I devote Chapter 9, "Getting Started," to getting connected.

  • Specifying where the ConnectionString should be persisted: The DSCW can save the generated (or selected) ConnectionString in the app.config file. Typically, this is what you'll want to do.

  • Specify the data source object: Basically, this means choosing one or more of the visible Table, View, or stored procedure objects exposed to the credentials supplied.

Of course, these are the same steps you have to take as you code a ConnectionString. What the DACW does not do for you is add exception handlers to deal with the litany of issues that can (and do) happen as you try to connect to a data source.

Let's get started by creating a new Windows forms (Smart Client) project. Once the project is created, walk through these steps to build a simple Data Source against the Authors Table.

1.
Open the Data Sources window by clicking on the Data | Show Data Sources (Shift+Alt+D) menu. Sure, it makes sense to dock this window in some convenient place.

2.
Click "Add New Data Source..." in the Data Sources window (or simply click the Data | Add New Data Source menu).

3.
This launches the Data Source Configuration wizard, shown in Figure 6.5.

Figure 6.5. The initial DSCW dialogchoosing the Data Source type.

[View full size image]

4.
Choosing "Database" tells the DACW that you want to set up a connection to a specific database and extract data from its data objects. Click "Database" and "Next".

5.
This launches the DSCW "Choose Your Data Connection" dialog. Here, you can choose one of the Data Connections already configured (as I discussed in Chapter 9) or create an entirely new Data Connection. Basically, what you're doing here is choosing a ConnectionString. In this case, I'm going to use a specific SQL Server login designed for this application type. It has limited rightsit's permitted to access only a couple of tables in the Biblio database and only one stored procedure. The configured dialog is shown in Figure 6.6.

Figure 6.6. Specifying a SQL Server instance and login credentials.

6.
For this example, I'm using a SQL Server loginnot SSPI (integrated) security. Because I specified that the password is to be persisted in the ConnectionString, the DSCW dialog reminds us that this is not such a good idea. It gives us another chance to remove the password from the ConnectionString. This dialog (shown in Figure 6.7) also permits you to view the ConnectionString to make sure it includes all of the parameters you need to specify.

Figure 6.7. Choosing (or adding) a data connection for the DACW.

[View full size image]

By not saving the SQL Server password in the ConnectionString, you're going to have to remember to pass it to the DACW-generated code before you'll be able to open the connection. This approach also assumes that you've enabled "mixed mode" security on the target SQL Server instance. I discuss how (and when) to do this in Chapter 9.

7.
The next dialog asks if you want to persist the ConnectionString in the application configuration file (app.config) and, if so, how to name it in the file. You'll take the default here and save it as "BiblioConnectionString", as shown in Figure 6.8.

Figure 6.8. Naming the persisted ConnectionString.

[View full size image]

Once a ConnectionString is persisted in the app.config file[4], you can reuse it for additional Data Sources by selecting the named Data Connection (MySettings). If you need to modify the ConnectionString, you don't (shouldn't) have to open the app.config file (once the DSCW is done). When working with Windows Forms, you can simply open the project property page and examine the Settings section (as shown in Figure 6.9).

[4] No, I'm not going to show you the XML app.config file. Take a look, though, if you're curious. Just don't touch anything.

Figure 6.9. Reviewing and editing the ConnectionString generated by the DSCW.

[View full size image]

Tip

ASP.NET programs don't support the DSCW. You'll need to use the Web project tools to change the web.config-persisted ConnectionString using a property page.

Here, you'll find the generated ConnectionString in a strongly typed application setting. To change it, click the ellipsis (...) in the Value column.

8.
Next, the DSCW prompts you to choose the objects to query (as shown in Figure 6.11). Since this login has limited rights, the DSCW "Choose Your Database Objects" dialog only populates itself with a list of database tables and views that the login has rights to access. In this case, our SQL Server login has been granted rights to the Authors, Title_Author, Titles, and Publishers Tables and the AuthorsByYearBorn stored procedureand that's it.

Incidentally, to deny rights to a selected database user (tied to the SQL Server login I'm using for this Data Source), I used the custom stored procedure DenyPermissionsOnTo, which is included in the Biblio database and in the support files included with the book. Basically, this stored procedure walks the objects in the current database and executes three T-SQL DENY statements on each object: ALL, ALTER, and TAKE OWNERSHIP. You must leave the CONTROL and VIEW DEFINITION permissions unset (neither granted nor denied) so the DACW can access (see) the objects. These permissions were set using SQL Server Management Studio, as shown in Figure 6.10.

Figure 6.10. Setting rights to the Authors table for a specific database user.

[View full size image]

Whatever you do, don't just arbitrarily check off any of these check boxes unless you understand what's going to happen. If you do, all of the objects in the list will be used to generate TableAdapter classes. If each TableAdapter takes over 850 lines of code, it might take until next June to get all the code generated.

Figure 6.11. The "Choose Your Database Objects" dialog.

9.
At this point, you can drill into the object lists by clicking the "+" sign next to the object icon. If there is no "+", the login does not have rights to any of this type of object. Clicking the "+" explodes the list and shows you the objects to which the login has rights to access. Remember, our login is allowed to see only a few tables and stored procedures, as shown in the list in Figure 6.12.

Figure 6.12. The exploded list of available database objects based on the login user's rights.

[View full size image]

10.
Before clicking "Finish", be sure to check the specific columns of the objects you wish to use to generate the TableAdapter and DataSet. No, don't select more columns than you actually need. You'll also discover that some DBAs restrict rights to specific columns. Individual columns might be set to deny UPDATE, DELETE, or SELECT (or other) rights.

If you choose an object that you don't have rights to view, you'll get an error dialog that warns you that you won't be able to access these objects. If this occurs, it's likely that the DBA has changed the rights for these objects since the list was generated.

In our example, I choose to access the Authors table and I named our DataSet accordingly. However, consider that a single DataSet might include data from several database Tables, Views, or stored procedure rowsets, so a more generic name might be more appropriate. The choice of this name is very important, though. It forms the basis of the strongly typed name used to reference all of the TableAdapter objects generated.

11.
Click "Finish" to start generation of the TableAdapter strongly typed class code and XSD XML for your Data Source. Once the DSCW is done, it displays a tree-view dialog that includes icons for each of the objects you chose, as shown in Figure 6.13.

Figure 6.13. The Data Sources tree-view control exposes the generated objects.

At this point, the Solution Explorer includes the new Data Source definition file, as shown in Figure 6.14. In our example, it's named dsAuthors.xsd. No, the XSD file is not deployed with your applicationit's used only at design-time to manage the auto-generated TableAdapter. If you ever want to delete a Data Source, simply delete this file.

Figure 6.14. The Solution Explorer (with Show All Files checked).

The generated TableAdapter code is stored in the dsAuthors.Designer.vb file. While you can review the code in this file, it's not wise to revise it. It's one of those "brain-surgery-in-the-mirror" things that can cause unexpected side effects. If you feel a need to make a change to your TableAdapter, I recommend that you try to use the Data Source configuration toolI'll discuss that next.


Configuring a TableAdapter

Our next task is to step through the process of modifying and configuring the TableAdapter. Generally, you're going to want to make changes to the generated code in almost every case. Let's look at several typical issues:

  • The DSCW does not permit you to create a T-SQL query of any kind when it's first constructed. For many situations, I prefer to JOIN two or more Tables or Views to return an SQL product. In this case, you'll need to use the Query Designer to build the SELECT statement.

  • Remember, the SELECT query generated returns only the specified columns (as shown in Figure 6.12), but the generated T-SQL does not include any means of limiting the number of rows returned. Unless you're creating a TableAdapter for a lookup table where you need all of the rows in a Table, you'll want to modify the WHERE clause or at least set a TOP clause to return N rows or modify the WHERE clause to include parameters to focus the query on a specific subset of rows.

  • In some cases, you need to execute a Function within the T-SQL used to return the rowset. I use functions (as described in Chapter 13, "Managing SQL Server CLR Executables") to manage "multi-select" pick lists.

  • You might also need to alter the order in which rows are returned by SQL Server. In this case, you'll need to add an ORDER BY clause.

  • In some cases, you might want to change (alias) the column names returned or reorder the columns so that they appear in complex bound controls (like a DataGridView) in a more readable fashion.

  • You might want to add T-SQL to the SELECT to add "All" or "None" selectors to lookup table queries. This can be done with a UNION query, as I'll illustrate later.

Yes, there are probably other reasons to reopen the Data Source and make modifications, so let's walk through that process.

1.
Start by opening the Data Sources window. I have ours docked to the Solution Explorer (as shown in Figure 6.14).

2.
Right-click on the Data Source icon of choiceyes, there might be several Data Sources to choose from. At this point, you're confronted with a set of options, as shown in Figure 6.15.

Figure 6.15. The Data Source options menu.

Depending on what you want to do, click on one of the option icons:

  • Add New Data Source... to reopen the DSCW from the beginning. This new Data Source can use an entirely different data source (database, Web Service, or business object) or use the same ConnectionString as your existing Data Source.

  • Edit DataSet with Designer to open the GUI Data Source editor. This is one way to modify or extend an existing Data Source. This option is used to make changes to the SQL queries, add parameters, fine-tune the columns, or simply add objects to the DataSet.

  • Configure DataSet with wizard... to reopen the DSCW from the object-selection dialog (as shown in Figure 6.12). Use this option to simply add another Table, View, stored procedure, or Function to an existing Data Source.

  • Refresh to repopulate the Data Source window from the stored XSD.

  • Preview Data... to open a dialog to run the Fill query associated with a selected DataSet. I'll walk through this a bit later.

3.
For this exercise, click on "Edit DataSet with Designer"it affords the most options and can address all of the issues I raised earlier.

4.
This opens the GUI DataSet designer. This tool is basically an XML editor designed to modify the DataSet's XSD file. If you've been following along, you should see something like Figure 6.16.

Figure 6.16. The GUI DataSet designer showing the "dsAuthors" DataSet.

Upon closer inspection, you can see that the DataSet includes a primary key (the Au_ID column), it returns three columns (Au_ID, Author, and Year_Born), the TableAdapter is named "AuthorsTableAdapter", and both a Fill and a GetData fetch method have been created (by default) in the TableAdapter. No, you did not do anything when executing the DSCW to affect these settings other than choosing the Authors Table and selecting the columns to return. All of the rest were generated by default.

Let's explore a bit deeper into the generated TableAdapter.

5.
Right-click on the Authors table in the DataSet designer (not on the Fill SQL). This opens yet another menu, as shown in Figure 6.17.

Figure 6.17. Exploring the DataSet designer options.

These options permit you to perform a number of operations on the targeted Data Source DataSet "table". Click on:

  • The "Add" submenu to open another menu that permits you to:

    • Add a new Query to the DataSet

    • Add a Relation to another table in the DataSet

    • Add a Key to the DataSet

    • Add a new Column to the DataSet

    I'll discuss all of these options later in this section.

  • Configure... to open the TableAdapter Configuration wizard (TACW). I spend quite a bit of time here, as the TACW permits you to make quite a few changes to the query used to return rows to the TableAdapter. That's where I'm going next.

  • The Cut, Copy, Paste, or Delete items, to manage the "Authors" table object in the designer. These options let you clone an existing table in the designer.

  • Rename to rename the table.

  • AutoSize to automatically resize the graphical objects in the designer. The designer surface is intended to permit you to manage several tables at once. It also shows the relationships between those tables.

  • Preview Data... to open a dialog (as shown in Figure 6.18) to capture any input parameters, execute the SELECT query, and display any returned rows. This is a way to test any of the Fill or GetData queries defined for the DataSet. Note that once you press the Preview button, the dialog returns the total number of rows.

    Figure 6.18. Previewing the data returned by the generated SELECT query.

    [View full size image]

Since you want to tune the query generated by the DSCW, click "Configure" from the right-click menu to open the TACW. This opens the wizard (as shown in Figure 6.19). Initially, the SELECT query shown is the T-SQL generated by the DSCW for your DataSet (and TableAdapter SelectCommand). Notice that it's missing the WHERE and ORDER BY clauses. I'm about to fix that (and a few other things along the way).

Figure 6.19. The initial TACW dialog.

[View full size image]

Note that the TACW dialog permits you to change the query manually (just type in the changes) or via the "Query Builder...". This launches a SQL query editing tool virtually identical to the Query Designer I've discussed in earlier chapters (See Chapters 4, "Getting Started with Visual Basic," and 5, "Managing Executables with the Server Explorer") and used in SQL Server Management Studio.

The Advanced Options... button opens a dialog that lets you choose how the TACW (and the DACW) generates the concurrency code for your query. The DACW assumes that you want to build an updateable DataSet, even in cases where you know the table is not updateable. In this case, I don't have update rights on the Authors table, so I'll have to disable attempts to build update, delete, or insert routines.

6.
Click on the Advanced Options... button to open the Advanced Options dialog (shown in Figure 6.20).

Figure 6.20. The Advanced Options dialog.

[View full size image]

By default, the TACW generates (or attempts to generate) Insert, Update, and Delete statements based on the SELECT. I want to disable this feature, as I don't have rights to alter this table in any way. Once I uncheck the first check box, the others become disabled. Disabling the generation of the DML action commands reduces the size of the dsAuthors.Designer.vb to 725 (from 853) lines of generated code (once these changes are committed).

If you are working with an updateable table, your only option when it comes to concurrency management is to either accept what the TACW generates or remove all concurrency checks from the action commands used to prevent collisions. I'll discuss the implications of each of these strategies in Chapter 12 (Updating).

If you're using a database-generated primary key (like an Identity column), default values, or computed expressions, you'll need to fetch the database-generated values after a new row is added or a row is changed. To enable this feature for RW tables, be sure to check this box.

7.
Click OK when you're ready to go on to the next step. This returns you to the initial TACW dialog (see Figure 6.19). I'm now ready to tune up the SELECT query.

Note that no changes are made to either the XSD file or the generated code until you finish the TACW.

8.
Click on the "Query Builder..." button to open the Query Builder. You should see a dialog as shown in Figure 6.21. I'm going to use this tool to alter the SQL used to fetch our authors.

Figure 6.21. The TACW Query Builder.

One of the easiest things to change is the order of columns returned. You want the query sorted by Year_Born, so you'll need to display this column first. There are a several ways to do this:

  • Don't do anything here in the query window but edit the column layout later when you build the UI. If you take this approach, you'll have to repeat the reconfigure process each time you work with the Data Source.

  • Drag the Year_Born column to the top of the list of columns in the second (criteria) pane.

  • Uncheck all of the columns from the top (column selection) pane and re-check them in the order in which they are to appear in the SELECT.

  • Feel free to choose any of these methods to reorder the columns.

9.
Next, I want to hide the Au_ID column. Again, there are a several approaches to take:

  • Again, you can choose to not do anything here in the query window and edit the column layout later when you build the UI. If you take this approach, you'll have to repeat the reconfigure process each time you work with the Data Source.

  • Uncheck the column in the top (column selection) pane. This removes the column from the SELECT, which is fine unless you're building an updateable query. In this case, leaving out the PK will hopelessly confuse the wizards, and it won't be able to generate the DML queries.

  • Uncheck the column's "Output" check box. This also hides the column and removes it from the SELECTwith the same caveats.

  • Go all the way back to the DACW column selection dialog and uncheck any column you don't want included in the SELECT. Again, leaving out the PK cripples the ability to generate an updateable DataSet/TableAdapter.

  • Use one of these techniques to hide the TimeStamp column. You don't need it.

10.
Next, you want to order the SELECT by Year_Born. Yep, there are a couple of options here as well:

  • As with some of the other changes I'm making, you can choose to defer the sort choice to the user at runtime or programmatically reorder the data using the Sort property of the DataView class. In addition, the DataGridView permits users to sort (ascending or descending) at runtime. You might find this approach yields better speed than having the rowset presorted on the server.

  • Add an ORDER BY clause to the Query pane. In this case, you simply add "ORDER BY Year_Born" or "ORDER BY Year_Born DESC" to the T-SQL in the query pane, depending on how you want the data sorted.

  • Click in the Sort Order column of the criteria pane and set Year_Born to "1". The Sort Type is set to "Ascending" automatically. Change this to "Descending", if desired.

11.
Next, I want to include a computed column "Age" in the query. This column returns a value generated by a T-SQL expression. Yes, this expression can call a SQL Server Function to perform the calculation if your login has rights to the function. However, in this case, the math is very simple, so I'll just hard-code it here (and I don't have rights to any functions). Again, you have a couple of options to set up this new column:

  • Add the "Age" expression to the SELECT. In this case, simply add "DATEPART(YYYY, GETDATE()) - Year_Born AS Age," to the SELECT.

  • Add the "Age" expression in a new row in the criteria pane. In this case, the DATEPART expression is entered in the Column column and "Age" is entered in the Alias column. The result of the changes I've made so far look like Figure 6.22yes, I tested the query, so the "Results" pane shows the rows returned from the query.

    Figure 6.22. The Query Builder is used to tune the SELECT query.

    [View full size image]

Adding Parameters to a Query

You still have a problemas you can see, the query is still returning 6263 rows. You really need to focus this query on a selected set of authors to get the number of rows returned to 50 or so. There are several approaches to this problem that can be solved right here in the Query Builder:

  • Add a TOP clause to the SELECT. This would limit the number of rows by number (TOP N) or by a percent (TOP PERCENT) of the rows that result from the other criteria. This approach might (would, in this case) hide important data, so its use should be governed by a thorough understanding of how TOP expressions work.

  • Add a WHERE clause that focuses the query on a name match (using an equality (=<parameter>) or a SOUNDEX or LIKE (%<parameter>) expression. You might also use an IN clause here, but note that you can't pass a parameter to an IN clause[5].

    [5] There are ways to pass a delimited sting via parameter to a SQL Function that generates a Table-type variable that can be few into an IN clauseI discuss this in Chapter 13.

  • Add a WHERE clause that focuses the query on a range of acceptable ages or Year_Born value(s) passed in as parameters.

  • Add a SET ROWCOUNT N clause to limit the number of rows returned. This works like the TOP clause, but even more arbitrarily. Unfortunately, this approach won't work with the Query Builderit works just fine if you're executing a stored procedure.

Whatever option you choose, you'll want to consider using an approach that permits you to bypass or combine the criteria tests so you can query for multiple matches. In other words, you might want to find authors who are 25 years of age whose name starts with "William" in one query and only 60 year-old authors in another querywithout changing the T-SQL. Implementing this option is simply a matter of "OR"ing and "AND"ing the logic together and including tests for NULL (or special value) parameters.

Tip

As you use criteria to limit the scope of your queries, be sure to deal with columns with NULL values.

Should all of this be done in a stored procedure? Sure, that makes a lot of sense. But when I'm prototyping, I often start from simple SELECT statements and work up from there to more complex ad hoc queries and from there to stored procedures.

I'm ready to add a few parameters to the queryand, yes, this can be done with the Query Builder, so let's step through the process of adding parameters. In case you didn't notice, as you make changes to the Query pane, the changes are reflected in the Criteria and Column Selection panes as welljust as changes to one pane affect all the others. You might find it (far) easier to make a minor change in one panebe assured that these changes are applied universally (assuming the syntax is correct and not too complex).

1.
I'll start with a set of query criteria that focuses the rowset returned from the query on a range of Year_Born values. Add the following WHERE clause to the SQL pane: WHERE (Year_Born BETWEEN @YearLow AND @YearHigh). The two parameters are recognized by the Query Builder, as they begin with the provider's (SQL Server's) parameter lead-in character ("@"). Note that this change is reflected in the other panes.

2.
Next, I want to (alternatively) be able to limit the rowset based on the author's name. To implement this criteria check, add the following LIKE expression to the Or... column in the Criteria pane in the Author row, as shown in Figure 6.23. Again, this change in the Criteria pane is reflected in the SQL in the Query pane and in the column selector pane.

Figure 6.23. Setting a parameter check in the Criteria pane.

[View full size image]

At this point, the Query Builder (as configured) should look something like Figure 6.24. There's only one last change I want to make before I start testing the query. Since our parameters force SQL Server to return a reasonable set of rows, perhaps it now makes sense to set an upper limit on the number of rows returned. To implement this criteria, I added a TOP(100) clause to the beginning of the SELECT statement.

Figure 6.24. The tuned SELECT query.

[View full size image]

I'm ready to test the new query (if you haven't done so already).

3.
To test the altered query (a really good idea), click the "Execute Query" button. This exposes a Query Builder dialog (shown in Figure 6.25) that's designed to collect the query parameters.

Figure 6.25. Capturing query parameters.

Notice that the parameters all default to NULL. Once you set a value, subsequent executions of the query using the Execute Query button default to the settings you supply on the previous run. Try various values in your querybe sure to enter values that don't make sense and see what happens. Make sure the rows returned make sense. Your UI code that captures parameters from the user (I'll get to that step later) must ensure that any parameters' values fall within acceptable range and conform to the correct typethe strongly typed code can go only so far.

When using a LIKE expression, be sure to put the parameter on the left of the "%" wildcard character. This permits the SQL Server query optimizer from using the column's index (if present) to perform the query.

You're finally done with the Query Builder, so click "OK" to commit the changes to the TACW. This should return you to the dialog shown in Figure 6.19. Your altered query should appear in the dialog. You're ready to click "Next".

Choosing Query Methods to Generate

When you create a new TableAdapter or when working with a pre-configured TableAdapter, you often need to revisit the methods the TableAdapter exposes to open the connection, run the query, and return the rowset. By default, the TACW generates two methods to perform (all of) these operations:

  • The Fill method, which executes the SQL you defined or a specified stored procedure and populates a DataTable or DataSet passed to the method. Sure, you'll get the chance to name this anything you like. Please don't. Give everyone a break and don't change the name of this method.

  • The GetData method, which executes the SQL you defined or a specified stored procedure and returns a new (system) DataTable. This method saves you the trouble of creating a new instance of the strongly typed DataTable and passing it to the GetData method.

The TACW also permits you to specify whether or not to generate the action commands (DML) to add, change, and delete rows. When you choose a stored procedure to execute, the option is grayed out, but you're still given an opportunity to specify stored procedures to perform these operations on the rowset returned by the "select" command (the Fill or GetTable methods) used to populate the DataTable. Let's walk through the query method configuration.

1.
Right-click on the TableAdapter in the Data Source designer (.XSD for the Authors table), as shown in Figure 6.20. You should now see the "Choose Methods to Generate" dialog, shown in Figure 6.26.

Figure 6.26. Choosing the rowset population and whether DML commands are generated.

[View full size image]

As promised, this dialog lets you enable the (renameable) Fill and GetData methods, as well as decide whether or not to get the TACW to generate DML commands.

Behind the scenes in the generated code, the GetData method simply calls the Fill method, as shown in Figure 6.27, so it does not make much difference which method is calledGetData returns the strongly typed DataTable, and Fill accepts the strongly typed DataTable as a parameter. While the dialog says that you can pass a DataTable or DataSet to the Fill method, the generated code accepts only a strongly typed DataTable as a parameter.

Figure 6.27. The generated Fill and GetData methods.

[View full size image]

Leveraging Stored Procedures

If you're working with a stored procedure instead of an ad hoc query, you'll first see the following dialog (see Figure 6.28) which lets you specify the stored procedures used to perform the action commands. If you want the TACW to generate ADO.NET Command objects to call these stored procedures when it comes time to update your data, then go ahead and fill in the "Bind Commands" dialog.

Figure 6.28. Binding Commands to existing stored procedures.

[View full size image]

Note that I've built a TableAdapter that uses four different stored procedures to "select" (fetch) the initial rowset, and three others specifically designed to add new rows, change existing rows, and delete existing rows.

It's bad if you mix up these stored procedure assignments.

As you select each stored procedure, you'll be given an opportunity to choose how the stored procedure action command parameters are to be populated (from the initial "Select" query. As shown in Figure 6.29, the TACW exposes a drop-down list to permit you to select any one of the input source columns. It's not usually necessary to choose an alternate valueother than disable passing any value at all to permit the stored procedure to use the default value.

Figure 6.29. Selecting the parameter mapping for the action Commands.

[View full size image]

When you click Next, (from the dialog in Figure 6.28), you'll be taken to the dialog shown in Figure 6.26, but the "Create Methods" check box is disabled.

When you click "Finish" in the dialog shown in Figure 6.28, the TACW generates (or rebuilds) the language-specific generated code that executes the selected stored procedures when the ADO.NET Update method is called on the TableAdapter. This generated code is written to a "designer" file named after the TableAdapter. In a Visual Basic .NET project, it would be called: dsAuthorsByYearBorn.Designer.vb. Figure 6.30 shows a snapshot of part of this (now 1311 line file). I'll discuss how ADO.NET manages the DataAdapter in Chapter 8.

Figure 6.30. The UpdateCommand generated by the TACW.

[View full size image]


Binding to the TableAdapter Using Drag and Drop

Once you "finish" with a TableAdapter configuration, you're almost ready to run the code that's been generated. As I step through the demos you've seen so many times, let's take the next step and drag a TableAdapter from the Data Sources window to a Windows Form. But waitbefore you drag anything from the Data Sources window, click the selected TableAdapter icon and click the down arrow. This exposes a menu, as shown in Figure 6.31.

Figure 6.31. Choosing how the TableAdapter is to be rendered when dragged to a Form.

By default, dragging a TableAdapter from the Data Sources window incorporates a DataGridView as the bound control. I'll start from there and re-visit how "Details" binding works a bit later.

1.
Open the Form "design" window. It can be customized as needed, but the process of dragging a TableAdapter pretty much takes over the Form, so other elements might be completely or partially overlaid.

If you don't like the placement of the controls placed on your Form, you can click Ctrl+Z to undo the drag-and-drop operation or simply rearrange the controls as needed.

2.
Open the Data Sources window and select the TableAdapter you wish to execute. Drag the icon for this TableAdapter to the Form. Wait a few seconds while the code generator does its job. The result should look something like Figure 6.32.

Figure 6.32. UI generated by dragging the Author's TableAdapter to a form.

[View full size image]

Ah, your UI might not look like this at first. I set the Form.Text property and resized the form to expose the third parameter (AuthorWanted) and the "Fill" (FillToolStripButton) button. No, I don't think this "button" looks like a button, either. But that's a bug in the UI generator, and I'll fix that in a minute. I also moved the DataGridView control up and to the leftcloser to the top of the form.

First, take a closer look at the components that have been added to your Form:

  • A BindingNavigator: Docked at the top of the Form (I assume your Form looks something like Figure 6.32), you'll find a customized BindingNavigator control (new for Visual Studio 2005), as highlighted in Figure 6.33. This control includes several functional elements:

    • Navigation controls: The first, previous, next, and last VCR-style buttons permit you to add code to move the current row pointer. If you double-click on these buttons at design time, you won't see any codeit's there though, hidden behind the scenes in the BindingNavigator control. It's also bound to the DataTable returned by the TableAdapter.

    • Row position and total rows count: The BindingNavigator control displays the current row pointer and the DataTable.Rows.Count property.

    Figure 6.33. The customized BindingNavigator.

    [View full size image]

  • An Add, Delete, and Save button: By default, these buttons are wired to the DataAdapter Update method built by the TACW. If you didn't choose to include code generation for these operations, the event handlers for these buttons must be populated with your own custom update logic.

  • The "Fill" ToolStripPanel: The ToolStripPanel (new for Visual Studio 2005) is simply a dockable container for the Fill Button control and any query parameters you've defined. Each parameter is exposed in its own labeled TextBox control. You're going want to edit the labels to make them readable as well as supply default values for the parameters.

  • A DataGridView control: Because I choose the default rendering option, dragging the TableAdapter to the form instructs the code generator to bind to a DataGridView (new for Visual Studio 2005). The control is placed at the cursor "drop point", so you'll probably want to reposition it or dock it to the Form.

  • A BindingSource control: At the bottom of the Form, you'll find icons for all of the controls added to the Form, including a BindingSource. This control (described in more detail in Chapter 7, "Managing Data Tools and Data Binding") binds a data source with one or more bound controls.

  • A generated DataSet: The drag-and-drop code generator exposes a custom DataSet named after and returned by the TableAdapter. This is bound to the BindingSource.

  • A TableAdapter: The drag-and-drop code generator also builds a custom TableAdapter for the form. This is used to expose the DataSet as a data source.

Customizing the Generated UI

Notice that the "Fill" button on the FillToolStrip does no look much like a Buttonor, at least, I don't think it does. To fix this, you can right-click on the "Fill" button and choose "Set Image" from the menu, as shown in Figure 6.35. Choose an appropriate image file and set the DisplayStyle to "Image" or "Image and Text". This will let your users have a better chance at finding the "Fill" button. No, I don't think it's a good idea to pre-populate the Form with datanot until the user has chosen the input parameters. This makes your Form load faster. Sure, if you save the user preferences, you might return to the last set of parameters and execute Fill when the form is loaded using those parameters.

Figure 6.34. The controls added to the Tool Tray area of the Form by drag-and-drop.

Figure 6.35. Customizing the "FillToolStrip" button.

Since I'm focusing on parameters, I need to make a few more changes to the generated codeat least, indirectly. I need to set the parameter default values (as I have already done in Figure 6.34). Click on each of the TextBox controls in the FillToolStrip and set the Text property. Where do these default parameters come from? Well, I like to "remember" the parameters last used, so I save the values in a persisted application setting that's reloaded when the application is restarted.

Using SmartTags to Manage Bound Controls

Each of the bound controls placed on the Form by the code generators can be further tuned by using the SmartTag exposed in the upper-right corner of the control. To see what's available for each bound control, I suggest you visit each control and click on the SmartTag and review what options are exposed. I clicked on the SmartTag exposed on the DataGridView to expose the "tasks" menu, as shown in Figure 6.36.

Figure 6.36. Clicking the SmartTag exposes additional "task" settings.

[View full size image]

Note that these "tasks" permit you to easily enable DML changes, column reordering, and docking options. I talked about using this approach to make UI (instead of query) changes earlier in this chapter. You can also choose to edit or add columns to the DataGridView using this interface. One change you might want to make is to make sure any RO columns in the DataGridView Columns collection have their ReadOnly property set to True.

1.
Click on the DataGridView SmartTag and click on Edit Columns... to expose the Bound Columns Properties page, shown in Figure 6.37.

Figure 6.37. Editing DataGridView Column properties via the SmartTag tasks interface.

[View full size image]

2.
Next, choose the column to edit and make whatever changes you feel are necessary.

Binding to "Details"

Just before I dragged the TableAdapter to the Form, I made a decision to use the default binding controlthe DataGridView. An alternative would be to choose "Details" from the menu shown in Figure 6.31. Let's see what happens when you make this choice.

1.
Return to the Data Sources window and click the Authors TableAdapter.

2.
Choose "Details" from the drop-down menu and drag the icon to an empty Form. Can you drag more than one TableAdapter to a form? Sure. You won't want to, thoughthe generated UI is not particularly pretty.

Figure 6.38. Choosing "Details" binds the TableAdapter data to individual controls.

[View full size image]

Binding to Custom Controls

Sure, you don't have to use a TextBox control to expose the contents of a TableAdapter column. You can navigate to the Data Source and click on each individual column returned by the query and choose the type of control to be used (as shown in Figure 6.39). By default, the code generators try to map appropriate bound controls to each column. This way, string and number columns are bound to TextBox controls, but pictures (TEXT, IMAGE, or VarChar(Max)) data types are mapped to Image controls. You can also code and specify custom bound controls. For columns you don't want users to change, you can set the bound control type to Labelat least, this works when binding using the "details" mode. Changing the type to "Label" has no impact on the RO status of the DataGridView column.

Figure 6.39. Clicking on individual columns determines how the column is to be bound.

Previous Page
Next Page


Using a TableAdapter DirectlyWithout Drag-and-Drop?

Okay, so I'm not a fan of drag-and-drop (D&D). While the code generation behind the scenes is far more sophisticated than ever before, it often adds a lot of overhead to the Form that I simply don't need or want. So, how can you use these TableAdapter classes without dragging it on the Form and having to deal with all of the...ah, "stuff" that gets added to the form? Once you've created a TableAdapter, it's exposed there in the Data Sources window. If you look closely at the Data Sources window, you'll see two different icons, as shown in Figure 6.40.

Figure 6.40. A populated Data Sources window with two TableAdapters.

  • The named DataSet icon (in this case, named "dsBiblio") helps identify the database table or "source" of the data. It's a direct reference to the ConnectionString and the DataAdapter used to manage the TableAdapter classes it holds.

  • The named TableAdapter icon contains the query definition and methods used to populate the underlying strongly typed DataTable. In this case, there are two TableAdapter classes defined: Authors and Titles.

The reason I'm mentioning this is that these names can help you remember the right naming conventions to instantiate and reference the objects. Running the Data Sourcegenerated strongly typed TableAdapter Fill or GetData methods is easy if you know the object class hierarchy. The TableAdapter is a member of the DataSet.TableAdapters collection. Each specifically referenced TableAdapter exposes a Fill method, a GetData method, and/or any number of custom methods. To make things a bit more difficult (but more flexible), each of these methods can be named just about anything. You might have to revisit the Data Source Designer to see how they have been designed.

So, how does this look in code? Take a look at Figure 6.41. Notice how I instantiate a new instance of the named DataSet's AuthorsTableAdapter and execute its Fill and GetData methods to return or populate either an untyped DataTable instance or a strongly typed AuthorsDataTable instance.

Figure 6.41. Addressing strongly typed TableAdapter and DataTable instances.

[View full size image]


Managing DataTable Classes in the Data Source Designer

When a Data Source .XSD file is exposed in the Data Source designer, each TableAdapter in the Data Source is displayed on a GUI surface. If the DSCW discovers that the TableAdapter DataTables have primary keys that match other tables, it generates appropriate Relationship instances in code and, when the XSD is displayed, the Data Source designer draws relationship lines between the DataTables, as shown in Figure 6.42.

Figure 6.42. A populated Data Source showing intertable Relationships.

[View full size image]

IMHO

You can arrange the individual DataTable graphics anywhere on the designersorta like a mini-database diagram, but without many of the features.

The relationship lines indicate that the TACW has generated DataRelation instances for each of these relationships. This is a new feature of Visual Studio 2005 that was missed by those working with hierarchical relationships in earlier versionsthey had to be hand-coded before. The question is, how does drag-and-drop deal with intertable relationships? The answer: It doesn't.

Setting DataTable Properties

This interface can also be used to set a few properties on the DataTable. Only a couple of these are interestingand one does not seem to work at all. If you right-click on a DataTable and choose "Properties", the Visual Studio property page focuses on the DataTable properties:

  • CaseSensitive: This property determines if the string comparisons on this DataTable's text columns use case-sensitive or non-case-sensitive comparisons. For example, if you wanted to set a WHERE clause parameter to "MAC" to find all authors whose name was "MACintosh" but exclude those spelled "Macintosh", you could set the CaseSensitive property to True. This (and the Locale properties) might have been more useful if they applied to a specific column instead of the entire DataTable.

  • DisplayExpression: There is very little known about this property. It seems to require a simple expression to set the value of a specific column. For example, I might have been able to set the Age column to "Age=2005-Year_Born". However, once I set this value, there was no sign that this did anything to the contents of the bound controls.

  • Locale: As with the CaseSensitive property, Locale determines the locale to use when comparing strings in the DataTable.

  • Name: This simply renames the DataTable in the TableAdapter.

The Generated Form Code

Most of the code generated by the drag-and-drop operation is created behind the scenes. However, this process also adds (at least two) routines to the Public Class code that you'll have an opportunity to modify or screw up. These code segments are shown in Figure 6.43.

Figure 6.43. Code generated by drag-and-drop in the Public Form class.

[View full size image]

The first routine (fired when the user clicks the "Save" icon in the BindingNavigator) ensures that any pending changes made to any bound controls are committed to the underlying DataTable (EndEdit) and subsequently call the Update method on the TableAdapter. This, in turn, calls the DML action commands defined in the TableAdapter. I showed you the generated code for the UpdateCommand in Figure 6.30.

The second routine captures the parameters from the Fill ToolStrip control and passes them to the TableAdapter Fill method. This method opens the Connection, executes the query, and populates the DataTable Rows collection with the rowset returned from the query.

Note that the exception handler generated is kinda sparse. I expect you'll want to flesh this out a bit for a production application.


Moving Data Sources Between Applications

So, what if you spend several hours building a fancy Data Source and you want to share this work with others on your team or simply include it in another application? It's simple. Follow these few steps to import an existing Data Source (.XSD) into your application.

1.
In the target project, create an Application Setting entry that matches the ConnectionString used in the .XSD file you're importing. Don't know how to name it? Well, that's easy, too. Simply open the .XSD file with any editor and look for the Connection element. It names the AppSettingsPropertyName that you need to re-create. See Figure 6.44 for an example.

Figure 6.44. Exploring a Data Source XSD file for the ConnectionString.

[View full size image]

2.
Next, add an "existing" item to your project and browse to the location of the .XSD file. When Visual Studio sees this file, it immediately compiles it to regenerate all of the code built for the source application. Your Data Sources window should populate with all of the top-level objects described by the XSD file.

3.
If you get a compile error when referencing the ConnectionString setting, modify the references to the setting to remove the hard-coded reference to the application name or change it to reflect the application's actual name.


Creating Web Service Data Sources

When the .NET Framework was first announced in 1999 and finally released in 2002, Microsoft made everyone believe that .NET was all about XML Web Services. It wasn't, and it still isn't. Because of this focus on Web Services, some developers working with traditional client/server or simple disconnected architectures were dissuaded from adopting the .NET Framework. Frankly, I think this is probably good. Because of .NET's low adoption rate, Microsoft has had the time to refine the .NET Framework and build up support for other architectures like Windows Forms (now dubbed "Smart Client"), Business Objects, and ASP.NET.

Today, Microsoft is still talking about Service Oriented Architectures (SOA), and Web Services is only a part of that approach to application design. Given the number of questions ordinary developers are asking about Web Services, I can only surmise that Web Services is playing no greater role than DCOM did before the .NET Framework. Because of this, I'm not going to focus heavily on Web Services (or Business Objects) here. I do want to outline the steps through the process of attaching a Web Service to a Data Source so that you don't have to face the frustration I did when trying this.

The DSCW can build a TableAdapter from a Web Service if the Web Service exposes a strongly typed data structure. No, the Data Source window does not simply expose the Web Services method(s), but the elements of the Web Service that return data. You'll also discover that the DSCW does very little for you beyond exposing these data elements. The Data Source designer is not implemented when working with Web Services, nor does it generate the code needed to call the Web Service method to return the data.

What Is a Web Service?

A Web Service is a block of code that looks alarmingly like an ASP.NET page that has no user interfacethat's because it is. Some feel that Web Services are designed to replace remote COM objects that run on a server somewhere on the LANwhat I call "distributed" COM objects or DCOM. Web Services have been used to implement SQL Server Reporting Services, SharePoint Services, Amazon.com e-commerce web services[6], and other sophisticated service engines. It's clear that one can use Web Services for some pretty serious and sophisticated applications.

[6] See www.amazon.com/gp/browse.html/104-8293193-5423957?%5Fencoding=UTF8&node=3435361.

IMHO

The fact that Web Services can expose objects, methods, and (most importantly) structured information over the Web makes Web Services very attractive to architects and developers wishing to increase the breadth of their applications.

When working with any object that runs in another memory space on another processor (like Web Services), you have to think about each operation the object is to perform in a number of different ways:

  • The object must be managed independently from the applications that make use of it. In the case of Web Service objects, it means that the IIS server that's hosting the Web Service is responsible for managing (and restricting) access to the servicejust as it would for any other ASP executable. IIS launches instances of the Web Service on demand and manages a pool of running (and waiting) instances. IIS prevents access by unauthorized users but permits administrators to fine-tune those restrictions.

  • The object must be visible to the LAN or the Web. That is, once you know where to look, you have to determine whether the object is present, accessible, and running. In the case of Web Services, you can query the target server (if you know its address) or depend on a "UDDI" server to expose registered Web Services to those who go trolling for services.

  • The object structure must be "discoverable." This means an application must be able to determine what the object "looks like" in its current state. For example, the tools and your application have to be able to determine what entry points the Web Service exposes (methods), what arguments are to be applied to those methods, and (most importantly for Data Sources) what data structure is returned. Since remote code can and does evolve over time, there must be a mechanism to ensure that the application is made aware that the signature of the object has changed.

  • The security context in which a Web Service executes is (usually) independent of the "user" running the application. As with ASP applications, managing security and data connections for Web Services can be fairly complex. Typically, an ASP application runs either under the ASPNET account or under a specially configured SQL Server Login account that is granted specific (limited) rights. Web Services are no different in this respect.

  • Web Services are different from ASP applications, in that they are designed to be controlled by and extrude XML result sets. To standardize (and popularize) Web Services, the Simple Object Access Protocol (SOAP) was developed and handed over to the XML Protocol Working Group of the World Wide Web Consortium. This means developers (usually via tools like Visual Studio) interact with Web Services via SOAP.

It's really beyond the scope of this book to try to teach you how to build and manage a professional Web Service or how to access remote objects using the SOAP protocol. However, I do know there are a number of data sources out there that are exposed as XML Web Services, and I do want to help you gain access to this data. Consider that the ReportViewer control can also be used to expose data from a Web Service. This makes it doubly important to make sure you understand how to leverage them and how to help others that do build Web Services for you understand how they must be configured so that your applications can access them.

Constructing a Sample Web Service

At this point, I want to walk you through the process of building a Web Service that returns data capable of being exposed by a Data Source, so I can use it as an example. No, not all Web Service data structures can be "seen" by the Visual Studio Data Source Configuration wizard (DSCW)only those that return strongly typed DataTable or DataSet objects. Remember, when the DSCW runs, it creates a strongly typed TableAdapter. In order for your Web Service to be visible to the DSCW, you'll have to return a strongly typed TableAdapter one way or another.

IMHO

The process I'm about to go through is made more difficult by the lack of tools on the ASP.NET side of Visual Studio.

1.
Start Visual Studio 2005 and click on "File | New Web Site...". No, I'm not really going to create a web site in the traditional sense, but remember that a Web Service is managed by IIS (or the local IIS host).

2.
Choose "ASP.NET Web Service" from the list of new web site templates.

3.
Name the web site so it's easy to find (and does not collide with another site). I chose to create the new Web Service on the local IIS system, but the template wizard can build the site on the file system, an FTP site, or a selected remote site. Click "Browse" to display a dialog to help locate the site (as shown in Figure 6.45).

Figure 6.45. Choosing the local IIS "location" for the Web Service.

[View full size image]

If you build the Web Service on the file system, you'll have trouble testing it later.

4.
Click "Open" to generate the code for the Web Service "site." This also generates the required template Service class and a sample "Hello World" method, as shown in Figure 6.46.

Figure 6.46. The generated Web Service and sample method.

[View full size image]

5.
It's useful to debug this sample code. It triggers a few hints and makes a few changes that enable debugging (by changing the web.config file). Let's walk through that process. Press F5 to start debugging or press the green start debug arrow.

When you develop a Web Service on the local IIS server, the program files are persisted (by default) in the IIS directory, so you might want to back up this directory when you do source-control management.

6.
At this point, a dialog appears that informs you that you can't debug without changing the web.config fileit automatically changes it if you press OK.

7.
The next dialog (see Figure 6.47) is an instance of Internet Explorer (IE) that exposes the service. Note that the "IIS" server in this case is implemented via port 2300. It also includes a link to run the Web Service method. More importantly, the page includes a warning to remind you to set the Web Service decoration to address the correct namespace.

Figure 6.47. Internet Explorer shows this page when navigating to the Web Service.

Farther down on this page, you'll find code examples for C#, Visual Basic .NET, and C++ to modify the Web Service decoration (as shown in Figure 6.48). These are examples of how to change the namespace used by the Web Service to be unique.

Figure 6.48. Examples of providing a unique Web Service namespace.

[View full size image]

8.
If you want to, click on the "Hello World" link. This launches another page (see Figure 6.49) designed to execute the code in the Web Service and return an XML rendition of the results. This page also contains the coded SOAP request and response to help you write a direct SOAP invocation of the Web Servicefortunately, you won't have to go to that trouble, as Visual Studio is going to do that for us behind the scenes.

Figure 6.49. The auto-generated web page used to invoke the Web Service method.

Clicking "Invoke" returns XML that represents the data returned from the Web Service (as shown in Figure 6.50).

Figure 6.50. The XML returned by the "Hello World" Web Service.

I'm ready to return back to our project and make some changes to the Web Service class decorations and implement a real method.

9.
Close the IE windows. This should return you to Visual Studio and the App_Code/Service.xx code block.

At this point, alter the Web Service decoration to point to the path or the HTML location of your file-based web service. I also don't want to name our Web Service "service", so I need to change the name in a number of places:

1.
First, change the name of the Public Class Service. I'm naming the Web Service AuthorsWebService. Note that Visual Studio helps in this renaming process (to a very limited extent, in this case), as shown in Figure 6.51.

Figure 6.51. Renaming the Namespace and Web Service class.

2.
Next, change the name of the Service.aspx in the Solution Explorer to AuthorsWebService.aspx. I also changed the name of the Service.VB file to AWService.vb, and changed the AuthorsWebService.asmx file to reflect these changes, as shown in Figure 6.52.

Figure 6.52. Renaming the Web Service support files to AuthorsWebService.

[View full size image]

3.
It would be a good idea to retest the Web Service (click the debug button) to ensure that these changes were made correctly.

IMHO

It's a shame that the Visual Studio Web Service template wizard did not give us an opportunity to properly name the service.

4.
I'm ready to add a strongly typed DataSet to the project. Remember that a Web Service can return only a strongly typed data structure.

Note that you won't find the Data Source Configuration wizard here, so you'll have to use one of the older "tried and true" methods to get the code generated or simply import the XSD file from a Windows Forms project.

5.
To build our strongly typed DataSet as I discussed earlier in the chapter, right-click on the project App_Code folder and select "Add New Item". This adds the code to generate a strongly typed DataSet to the project. I named this new DataSet AuthorsData. Click Next to launch the (now familiar) TableAdapter Configuration wizard.

6.
Step through the TACW to create a TableAdapter against the data you wish the Web Service to return. In this case, I execute the AuthorsByYearBorn stored procedure for our example Web Service and choose the GetData method to expose the data. The TACW creates an AuthorsByYearBorn.xsd file in the App_Code folder in our project.

7.
In the project, I need to instantiate and reference the strongly typed data class, so I add code to declare two new objects:

  • A strongly typed TableAdapter to expose the GetData method

  • A strongly typed DataTable to serve as a transport mechanism for the data returned from the Web Service

The code to implement these operations is shown in Figure 6.53.

Figure 6.53. Implementing the Web Service method to return a strongly typed DataTable.

[View full size image]

Note that I added the new method AuthorsByYearBorn (with the decoration of <WebMethod()>. This means our Web Service exposes two methods, but only one returns a strongly typed data structure.

The code also includes a subroutine to validate the input parameters to ensure that their range is correct. This is an example of a business rule of sorts. This Validate routine is shown in Figure 6.54.

Figure 6.54. Validating the input parameters in a business rule.

[View full size image]

8.
The new Web Service is now ready to test. Press F5 or the debug button to start the test. IE now exposes the AuthorsWebService and the single method I implemented as shown in Figure 6.55.

Figure 6.55. IE exposes both methods of our Web Service.

Tip

If you get an exception that complains that you don't have permission to view the page, make sure that the .ASPX page is set as the start page.

9.
Click on the circled "AuthorsByYearBorn" link in IE to launch the Web Service test application. Enter values for the two input parameters as circled in Figure 6.56, and click Invoke to execute the Web Service code that retrieves the data.

Figure 6.56. Enter the input parameters into the IE test page.

[View full size image]

To debug the Web Service code, simply set a breakpoint on any line of code in the Web Service language file (AWService.VB, in this example). When IE launches the Web Service, Visual Studio breaks at that line (if the logic reaches it).

At this point, IE displays the XML returned by the Web Service. It includes both the strongly typed schema and data from the query, as shown in Figure 6.57.

Now that your Web Service is tested, you're ready to access and test it from a "consumer" application.

Figure 6.57. The XML data returned by the Web Service.

[View full size image]

Consuming and Testing Web Services

I'm ready to use the Data Source Configuration wizard to locate, link to, and test our new Web Service. I usually start a new instance of Visual Studio to consume remote objectsit just makes managing the two projects simpler.

1.
Create a new Visual Studio Windows Forms project. I called ours ConsumeTestWebService.

2.
Create a new Data Source for the project by launching the DACW. Click on "Web Service" when asked to choose a Data Source Type. Click Next and you should see the "Start Browsing for Web Services" dialog, as shown in Figure 6.58.

Figure 6.58. Browsing for Web Services.

[View full size image]

3.
Click on "Web services on the local machine". Yes, you can enter the URL for the target Web Service if you know it, or ask Visual Studio to search for UDDI Servers (Web Service publishers) on the local network. This should expose not only the Web Service I just created (AuthorsWebService), but others created by members of our development team and those exposed by other Web Service providers. In this case (as shown in Figure 6.59), I can see Web Services exposed by Reporting Services 2000 and 2005.

Figure 6.59. The Web Service browser found these Web Services on the local system.

[View full size image]

4.
Click on the "AuthorsWebService" link to set the "Add Web Reference" dialog URL to point to our IIS server's AuthorsWebService.ASMX file and expose both of the Web Service's methods (as shown in Figure 6.60).

Figure 6.60. Choosing a specific Web Service exposes the methods it implements.

[View full size image]

5.
If you click on either method, you're taken back to the test routine you saw when you were developing the Web Service. Instead, click on the "Add Reference" button and "Finish" on the next dialog to confirm the Web reference URL I want to pass to our project and Data Source.

When the DSCW closes, the Visual Studio has generated no XSD file. Note that the second method exposed (HelloWorld) is not exposed in the Data Sources window at allit does not return a strongly typed data structure. But I'm not done yet. Visual Studio won't generate the XSD file or other important Web Service .disco and .wsdl files until you build the project.

6.
Right-click on the Project Explorer and choose "Build Project". This adds a number of files to the project as Visual Studio generates the TableAdapter for our Web Service Data Source. The final result is shown in Figure 6.61.

Figure 6.61. The Web Service "Data Source" and the post-build generated files.

I'm not sure you could really call this a "Data Source," as most of the functionality is not implemented for Web Services, but consider that the data definition is not here in this application as it is for the other Data Sources I've created so farit's in the Web Service. At this point, I'm permitted to consume the Web Service but not alter it or the data structures it returns. I can't add to or change the methods or their signatures, either.

Exposing the Web Service Data Source in the UI

Yes, you can drag and drop the Web Servicegenerated DataTable to the Form, and, yes, Visual Studio generates some of the same UI elements it created for a "Database" Data Sourcebut not all, and it does not add any code to Form_Load. The code is all up to you.

Dragging the AuthorsByYearBorn DataTable from the Data Source Designer to the Form generates (as shown in Figure 6.62):

  • AuthorsData: A strongly typed DataSet.

  • AuthorsByYearBornBindingSource: A BindingSource control to interface the data source and the bound control.

  • AuthorsByYearBornBindingNavigator: A UI element to navigate from row to row in the bound BindingSource data source.

  • A DataGridView control to display the data. This control is populated with the column names from the strongly typed DataSet.

Figure 6.62. Dragging the Web Service DataTable to a Windows Form.

Unlike a "database" Data Source, when working with a Web Service Data Source, you'll have to add code (by hand) to call the Web Service method(s) to populate the strongly typed DataTable. In this case, this means you'll have to supply two input parameters and code your own parameter-collection code to capture and validate them. You'll also have to code your own "Fill" button to execute the method code. It would be nice if Microsoft had included the "ToolStrip" button to manage these UI elements as it did for database Data Sources.

In about 5 minutes, I was able to create my own ToolStrip to handle each of the parameters and expose a button to launch the Fill (as shown in Figure 6.63).

Figure 6.63. The customized ToolStrip control.

The code to fetch the data is not all that complicatedonce you figure out where it's coming from. That part is not at all intuitive until you've created dozen of these applications. The surprising part is that the Web Service is exposed off the Form's (My) WebServices. The AuthorsWebService exposes each method (even HelloWorld). Intellisense helps you figure out how to build and reference the other objects that are passed and returned. Note that the input parameters are drawn from the ToolStrip TextBox controls. Since the strongly typed DataTable inherits from System.Data.DataTable, it's possible to bind to its DefaultView and sort the rowset before showing it to the user.

Figure 6.64. The code used to fetch execute the WebService method.

[View full size image]

I hope this chapter gives you a better understanding of the Data Source paradigm. I also hope you have enough information to help decide where this approach makes sense and where it doesn't.


Summary

This chapter toured the new (and numerous) wizards and tools used to generate code and populate the newly incarnated TableAdapter. As you can probably tell, I'm not totally convinced that the TableAdapter is a universal solution for data access. I don't think Microsoft believes it is either. I expect we agree that it's an important tool when building simpler data access applicationsand many of the applications we build fall into this category. I've also found that the TableAdapter is ideally suited for use against the SqlCE SQL Server Everywhere database. That's because SSEv databases are designed to be (predominantly) single-user so many of the collision issues that the TableAdapter has a tough time with are avoided. I also think that over time the TableAdapter will be replaced again (and again) as Microsoft tries to build more universal solutions for specific problems and markets. The next chapter wanders into the darker recesses of the Visual Studio data tools. Stick togetherI don't want anyone to get lostand kid, don't drink out of that puddle!


Chapter 7. Managing Data Tools and Data Binding

    Introduction

    Touring the Visual Studio Toolbox

    Using the DataSet Toolbox Element

    Introduction to Data Binding

    Using the BindingSource Class

    Using the BindingNavigator Control

    Using the DataGridView Control

    Using the ProgressBar Control

    Summary


Introduction

Visual Studio 2005 includes a number of new controls in the Toolbox that you need to understand and learn to leverage if you expect to be able to create professional applications. Of course, the Toolbox is stuffed full of controls and wizard-launchers that folks new to Visual Studio are also going to have to learn. It's well beyond the scope of this book to detail them all, but there are a number of data management controls that I would like to discuss in this chapter. When used correctly, some of these "data" controls can save you a great deal of time when building your application, and they can also be used as foundations for your own custom controls. That is, you can use Visual Studio to create your own custom composite controls that inherit from the existing controls. This way you can add events and properties, and interrelate two or more controls into a single custom control.

As usual, the Visual Studio team has reinvented too many of these controls for my liking. Long-timers tell a story about new developers at Microsoft. One of the first tasks these "kids" take on is to write a new data grid control. As a result, Microsoft is hip-deep in data grids of every conceivable typeand some are virtually inconceivable. Apparently, the best of these is dubbed the new Visual Studio data grid control each time Microsoft ships a new version. This time, it's the DataGridView. I'll discuss this control to some extent and walk through some of its new features and issues. Yup, it has lots of shiny new knobs, lights, and switches, but how many times can Microsoft re-create the shovel?

Controls and how they're accepted by the developer and third-party community are critically important to Visual StudioBill Gates thought so, and so do I. The very first version of Visual Basic was held back by Bill (the other one) as the Microsoft development team did not include "extensibility." That is, he wanted Visual Basic programmers to be able to buy third-party controls to extend the power of Visual Basic applications they wrote. I think this was a critically important decision and made the difference in how quickly Visual Basic has been adopted over the last decade. Without these third-party grids, custom dialogs, and other controls Microsoft was unable or unwilling to create and support, Visual Basic developers would have been stuck with the functionally challenged controls that shipped with Visual Basic. This extensible architecture also unloaded the Microsoft Visual Basic team so they were not pressured to create and support a host of world-class controls. Given their limited manpower and budget, Microsoft knew that third-party developers could spend a lot more time and talent on control development than they could. The result was the birth of an entire industry of control vendors.

IMHO

Remember that Visual Studio is an application development platform and is designed to build a number of different types of application architectures. Yes, both the Windows Forms and ASP.NET architectures support use of Visual Studio Toolbox controls to populate your UI. However, these two architectures are radically different, so it's unreasonable to expect them to behave the same or support the same functionalityat any level. Any similarity is an accidental blessing.

To leverage Visual Studio's application architecture flexibility and extensibility, several third-party companies have built controls to use with Windows Forms and ASP.NET application architectures. Some of these controls are distributed as free samples (like the terrific Windows Forms Validator controls written by Bill Hollis[1] that are included on the DVD). Others that I have worked with or seen demonstrated from time to time are distributed by companies like ComponentSource[2], (Dan Appleman's) Desaware, Apex, VideoSoft, and Dundas[3]. In any case, I recommend that you not try to launch any substantive project without first spending a few days exploring the library of controls already available in the industry. Sure, some of these controls can cost $50 or $1,000, but if you factor in the cost to duplicate even 50% of their functionality, most are well worth the one-time cost. Pay special attention to controls that supply the source code. This makes them far simpler to extend to your special needs.

[1] See www.dotnetmasters.com. Look under "Samples".

[2] See www.componentsource.com/ and www.desaware.com/index.aspx.

[3] The DVD includes free samples of most of this company's wares.

However, this chapter is more than a tour of the Visual Studio Toolbox. It introduces a new concept to Visual Studio developersthe strongly typed TableAdapter. As you'll quickly discover, I think this new technology has its place in some applications but is not a universally applicable data paradigm. Peter calls this the "over-hyped" adapterI think you'll see why he (and I) feels this way as you progress through these pages.


Touring the Visual Studio Toolbox

Visual Studio 2005's Toolbox is a far cry from the skimpy tools that shipped with Visual Basic 1.0 back in 1991[4]. It includes a number of new controls, refinements of existing controls, and replacements for others. You might also be disappointed (as I am) that some controls and wizard-launchers are missing from the Toolbox. For example, the shortcuts to the DataAdapter Configuration Wizard and other ADO.NET data access elements have been shelved. In a minute, I'll show you how to re-enable these deprecated data access controls.

[4] You might find Billy Hollis's History of Basic an entertaining overview of how I got here.

By this time, the Visual Studio 2005 Toolbox (shown in Figure 7.1) should be familiar to Visual Basic or C# developers. I've exploded the Data tab to expose the standard set of data-centric controls. Notably missing from the Data tab are the mainstay DataAdapter and other ADO.NET controls we all use so often. It's not hard to bring back these controls, as I'll show you next.

Figure 7.1. The Visual Studio 2005 Windows Forms Toolbox.

Exposing Deprecated Data Access Controls

Microsoft seems to have taken a moralizing stance on some of the new data access features. For reasons only they understand, they have chosen to "hide" some Visual Studio features that many of you have been using from the beginning. For example, every single ADO.NET book, article, or course discusses how to create an updateable DataSet with the DataAdapter Configuration wizard (the DACW). As I said, you won't find this wizard in the Visual Studio 2005 Toolbox. The "Command" and "Connection" wizard icons for each of the providers are gone, toolike the SqlCommand and SqlConnection wizard launchers.

Thankfully, this functionality is still in Visual Studio (for now). I expect they've been hidden because Microsoft doesn't want you to use themor cut yourself while running with scissors. I also expect they want developers to use the new Data Source Configuration wizard and strongly typed DataSet objects. These deprecated items in the Toolbox generated untyped DataSet objects. Many of us still find a number of good ways to use these productivity aidsespecially when teaching folks how to use ADO.NET. Yes, I expect there are still lots of mainstream developers who still depend on them and will miss them as soon as they start using Visual Studio 2005.

Thankfully, it's not hard to add these hidden controls and wizard shortcuts back to the Toolbarjust follow these steps:

1.
Start Visual Studio and create a Windows Forms project. Sure, this can be done in any languageeven C#.

2.
Pin down the Toolbox tab, as you're going to create a new tab.

3.
Right-click below the items in the Toolbox and click "Add Tab". Give your new tab a meaningful name, like "Deprecated Data Components That Microsoft Doesn't Want Me to Use"or perhaps something shorter, like "Untyped Data".

4.
Right-click again in your new tab and select "Choose Items...". This exposes the "Choose Toolbox Items" dialog (after a long wait while it's initialized).

5.
Scroll down to the section for your data access provider. For SQL Server, you'll want to add items from the list of SqlClient components. If you type "S", the pointer moves promptly to the first "S". You can also enter "SqlClient" in the Filter dialog to locate all matching components, as shown in Figure 7.2.

Figure 7.2. The Choose Toolbox Items selector dialog.

[View full size image]

6.
Repeat this process for any ODBC and OLE DB components that you want to expose in the Toolbar. When you're done, you should have a new Toolbox tab with the ADO.NET components you've used for the last three versions (as shown in Figure 7.3).

Figure 7.3. The Toolbox with the missing ADO.NET 1.1 components replaced.

Unfortunately, not all of the productivity features we all got used to in earlier versions of Visual Studio are present in Visual Studio 2005 and can't be added back. For example, when you want to create a quick call to a stored procedure and build the Parameters collection, you can (in Visual Studio 2003) drag a stored procedure from the Server Explorer to the Form, and Visual Studio takes care of the rest. Sadly, this does not work in the 2005 version. I understand that one of the new third-party refactoring add-ins can build a Parameters collection from a stored procedure, so all hope is not lost.

Finding the Right Control

One of the problems with having so many controls in the Toolbox is trying to find the one you want. Visual Studio doesn't help much when it comes to locating a specific control, as it supports only ascending or descending alphabetic sorting. However, I find it easiest to simply create your own "MFU" (most frequently used) tabas you choose controls for your project, simply drag the control to your own MFU tab with the Ctrl key held down (to copy the shortcut). Next, use the "Move Up" right-click option to move your custom list to the top of the Toolbox, or simply drag it there.

Before you can access the Toolbox, you'll need to open a Form window.

Understanding the Tool Tray

Some of the controls you'll find in the Toolbox are simply UI elements that expose a graphic on the Windows (or ASP) Form, as well as properties, methods, and events. Others are shortcuts that launch a wizard (like the DACW shortcut I exposed in the previous section). As you well know, to add these controls to your project, you either drag them to the Form or double-click them in the Toolbox. This operation instructs Visual Studio to generate code in Formx.designer.vb/cs. For most (but not all) controls, a visual UI element is placed on the form. For the shortcuts and controls that have no UI elements (such as the Timer, BindingNavigator, and the data access wizards), Visual Studio places an icon in the Form's Tool Tray (below the Form layout area, as shown in Figure 7.4). The icons are also used to set the properties of the controls or restart the wizard that placed them in the Tool Tray. Deleting the icon removes the project code generated to support them. However, it's always a good idea to understand the intercontrol relationships before deleting any cross-dependent control.

Figure 7.4. The Form's Tool Tray populated with several controls.

Tip

You can back out some drag-and-drop operations by deleting selected Tool Tray icons.

All this is basically unchanged from earlier versions of Visual Studio, so let's get to the new stuff.


Using the DataSet Toolbox Element

While the DataSet class is not new for Visual Studio, the DataSet element in the Toolbox is new. The DataSet Toolbox element launches yet another wizardin this case, it gives you two fundamentally different ways to build or reference a DataSet, as shown in Figure 7.5.

Figure 7.5. Adding a DataSet to a Form using the DataSet Toolbox element.

[View full size image]

In either case, this technique is used to add an XSD-based, strongly typed DataSet or untyped System.Data.DataSet DataSet instance to a Form or custom component without generating bound UI elements. Once the DataSet is added to your form, it becomes part of the Form code scope, so it appears in other dialogs that permit easier binding (as you'll see later in this chapter).

  • A Typed DataSet: In this case, the DataSet Toolbox element generates a named instance of a specified DataSet already in your project. The Name drop-down list enumerates the existing XSD strongly typed DataSet classes already created in your project. The documentation says you can simply enter a name that will be available at runtime. Apparently, this was not hooked uponly existing XSD DataSet classes were accepted. I see this element is useful when you want to place a reference of your XSD STD on the Form but don't want any UI elements generated, as happens when you drag and drop selected TableAdapter objects from the Data Source window.

    When you click "OK" on the Add DataSet dialog, Visual Studio adds an icon to the Tool Tray at the bottom of the Form's design-time UI. This element includes a "smart tag" (as shown in Figure 7.6) that can be used to launch the DataSet Designer. No, you can't drag this element to the Form.

    Figure 7.6. The Tool Tray DataSet icon supports a smart tag.

  • An untyped DataSet: In this case, Visual Studio simply creates a new named instance of System.Data.DataSet. The default name is "DataSet1," but you can rename it via the properties dialog. It won't be of much use unless you write the code to populate the schema and data. Binding to an untyped DataSet at design time is problematic, at best. Until Visual Studio can determine the schema (by running the query or inspecting the SELECT), it won't be able to help much with data binding. That's why there's so much emphasis on strongly typed DataSet classes. I can't see a practical use for this unless you need a named DataSet to bind to other controls at design timeexpecting to populate the other Properties at runtime.


Introduction to Data Binding

Before I wade into the Toolbox controls used to manage data binding, let's discuss a few points on data "binding" that can make the process of knowing how and when to use these Toolbox controls a bit easier. Since there are better, more in-depth sources of information on data binding,[5] I'm not going to go very deep here, but I hope to clear up some of the fundamental mysteries. The binding mechanism implemented by the Framework is another one of those generic blocks of code designed to bind to virtually anything from arrays to small farm animals with parental consent. To stay focused, I'm going to stick to binding to ADO.NET classes like the DataTable and the DataView it exposes.

[5] I reviewed and liked Brian Noyes's book Data Binding with Windows Forms 2.0 : Programming Smart Client Data Applications with .NET (Microsoft Net Development Series) (Addison-Wesley, 2006; ISBN 032126892X). Peter and I agree that it's a must-read for those who need more details on the inner workings of data binding.

Binding simply means connecting one or more control properties with data source valuesin other words, showing stuff from the database and posting the changes made in the UI back to the data source so it can be posted back to SQL Server.

The .NET Framework binding functions include posting incoming data to specified bound control properties (like the TextBox.Text) as it arrives at the data source (usually a DataTable or a TableAdapter, but a DataReader can be bound in ASP.NET applications, at least to some extent), refreshing the control(s) when the data source changes, and saving the changes back to the data source when your code determines that the user is done editing.

Ever since Visual Basic V1.0, Microsoft (and control vendors) has been struggling with data bindingit's been a tough nut to crack. The BindingSource control and the classes that support it behind the scenes are the latest attempt at an easier-to-use binding interface. Typically, these binding classes add another layer between the sources of data like TableAdapter, DataTable, and DataView instances, and more specialized data sources such as ArrayList instances.

For ASP.NET applications, data binding has improved quite a bit, in that it now supports full-duplex binding. That is, when data binding is half-duplex, it means that data flows from the bound data source (the DataTable) to the bound controls (like a DataGridView or TextBox). However, changes to the bound controls are not automatically posted to the DataSource. In ASP 1.1 and earlier versions, developers had to write code to move changes out of bound controls to the underlying "bound" data source. In 2.0, the Framework binding infrastructure supports full-duplex data binding. This means changes committed to bound controls are automatically posted to the data source.

IMHO

Based on the number of people who still get confused about data binding, I'm of the opinion that while the new data binding interface seems well received by those building tools, the average developer may indeed find this overwhelmingI did. To make matters worse, it's not that well documented in MSDN. All of the examples seem to work with object and array binding, which must be interesting for some, but hardly what most data access application developers will want to do. I'll try to simplify this discussion as much as possible, but I think you'll find that simply letting Visual Studio drag and drop generate the code might be the path of least resistance.

Complex Versus Simple Binding

When I bind to a control that can display only a single row from a single column, I use "simple" data binding. In this case, you typically bind to the visible property of the controllike the Text property of a TextBox. However, you can bind to any property of any control or to other objects as well. With simple binding, you have to manage "currency." That is, the control(s) bound to a data source (like a DataTable) can show only one row at a time. It's up to your code to position the current row pointer, know when you've reached the end or beginning of the rowset, and expose controls or other mechanisms to permit the user to navigate through the rowset. As I discussed earlier when demonstrating drag-and-drop data binding, Visual Studio automatically generates all of this code for you, if you let it.

Tip

You might use drag-and-drop binding to show what the underlying data binding should look like.

When I bind to a control that's capable of displaying an entire rowset, I use "complex" data binding. In this case, I bind to the control's DataSource property to the data source (like a DataTable) and DataMember property to a specific column if the control shows only one column. For example, to bind to a DataGridView in a Windows Forms application, you simply set the DataSource property to the DataTable or DataView. However, when binding to a ListBox or ComboBox control, you must also set the DataMember property to the specific column to displayby name. This means you'll have to hard-code the column's name as a string into the DataMember property. Complex bound controls are responsible for completing rowset populationbut that's up to you with simple bound controls. This means complex bound controls read all of the rows from a DataReader or simply pull all of the rows out of the DataTable to populate its internal display elements. When binding to a DataReader (in ASP.NET), once rowset population is complete, the DataReader is closed. Nope, DataReader binding is not an option for Windows Forms applicationsyou'll need to load a DataTable from a DataReader beforehand and bind to the DataTable.

Each Control object implements a collection of bindable elements (IBindableComponent), which permits you to bind any of the control's properties to a data source. An interesting use of this feature would be to bind a TextBox control's Text property to a DataTable column value, and the ReadOnly or the ForeColor properties to other data-driven columns to prevent the user from changing the valueand seeing that it's read-only based on the text color. Yes, each bound value can come from a different data source.

Let's review the control properties that are used with data binding.

  • DataSource: This property points to the "source" of the data. In data access applications, this property is usually set to a populated DataTable or the DataView returned by the DefaultView property.

  • DisplayMember: When working with a complex bound control that displays only one column (like a ComboBox or ListBox) or a simple bound control, you must set the DataMember to address a specific column in the data sourceby name.

  • ValueMember: In some cases, I bind to a DisplayMember like "StateName" and a ValueMember like "StateCode" that contains the encoded value that corresponds to the display member. This way, users can see and select an English word like "Washington" while the underlying code uses the ValueMember value "WA" behind the scenes.

As the .NET Framework has evolved, Microsoft has further refined data binding so that as developers' skills evolved, they would continue to find easier ways to connect data with bound controls. The 2.0 Framework has taken a number of steps to make things simplersome of the more complex interfaces were hidden behind new objects like the BindingSource, and additional options were added to accommodate more sophisticated designs. IMHO, it's still pretty complicated, but the designers and drag-and-drop can eliminate all of this code for you if you still can't figure it out.

Each .NET Framework control also implements a DataBindings collection. This collection can be populated in code (as shown in Figure 7.7). In this case, I added a DataBinding to each TextBox controlnaming the TextBox property I want to bind, the data source (a DataTable in this case), and the DataTable column's name.

Figure 7.7. Coding simple binding from a DataTable to a set of TextBox controls.

[View full size image]

The TextBox property and column name strings are not checked at design time. If you misspell any of the string parameters here, you're pooched and won't find out until you reach this code at runtimeif you do.

Remember, a control can have more than one property bound to a data source. However, the Framework won't let you define more than one binding to the same property.

Another approach to setting the DataBindings property is to use the Visual Studio UI property dialog (shown in Figure 7.8).

Figure 7.8. Choosing a binding Data Source.

1.
Click on the TextBox or other control you wish to bind to. Yes, this can be any control, including the Form itself.

2.
Press F4 or open the object's property page.

3.
Click on "(DataBindings)" in the property page, and then click on the ellipsis (...) after "Advanced". This opens the Formatting and Advanced Binding dialog (as shown in Figure 7.8).

4.
Select the control property to bind using the drop-down list (as shown in Figure 7.8) and navigate to the correct Data Source. Yes, there can be several of these as your project increases in complexity. Make sure you choose the right one.

5.
Next, set any property formatting and whether or not you want the bound control to attempt to update the data via a visible TableAdapter Update method (as shown in Figure 7.9).

Figure 7.9. Data binding using the Visual Studio Properties page.

[View full size image]

6.
Click "OK". At this point, the Visual Studio code generator takes over. It adds code to the Form Load event handler, which generates the following three controls and adds them to the Tool Tray on your form:

  • A named DataSet instance that addresses the selected Data Source

  • A named BindingSource instance (as I discuss later in this chapter)

  • A named TableAdapter instance

I discuss the generated code in the next section, when I tour the BindingSource class.

Frankly, this process was pretty tedious, as I had to repeat it for each and every control to bind. Behind the scenes, this GUI "RAD" process simply generates the same code I just built above (more or less)using cut and paste. Another issue with this approach is that if you don't choose the right data source (and there can be many to choose from, even in a simple application), you won't see the right data in your bound controlsif you see any at all. The benefit to this approach is the ability to set the property format. Sure, that can be set in code as well, if you need that additional functionality.

A disadvantage to this approach (in my mind) is the fact that when you use the UI to set up data binding, by default the Fill method is called in the Form Load event, so the user has to wait while the query is executed before the Form is visible. When you commit to this type of binding, the Visual Studio code generator writes a TableAdapter Fill call to the Form Load event, as shown in Figure 7.10.

Figure 7.10. Visual Studio adds code to your Form_Load when you use Property page data binding.

[View full size image]

Yes, you can (and probably should) remove this line of code if you manage query execution yourself. This helps your Form load faster and gives your user the opportunity to set input parameters for the Fill query.

IMHO

The sample application ("Binding Source") on the DVD illustrates these points in a complete example.

So what's missing? This seems pretty simple so far. Well, consider that I'm working with "simple" bound controls, and there is a list of other tasks that must dealt with somehow. Again, if you use drag-and-drop (D&D) data binding, many of these issues are handled automatically by generated code. If you find that this approach can't work in your situation, you'll need to take care of the following details that D&D handles automatically:

  • Initial rowset population: D&D creates the needed DataSet and TableAdapter instances and generates code to run the Fill or GetTable methods.

  • Binding: D&D generates the needed binding classes and sets the appropriate data binding properties (as shown in Figure 7.8).

  • Navigation: Whether binding simple or complex controls, D&D generates a BindingNavigator that includes navigation buttons. The BindingNavigator is tied directly to a CurrencyManager class (for list and rowset data sources) that provides the "move" methods to navigate between rows. I discuss this class later in this chapter.

  • Query parameter management: If the query you're executing uses input parameters, the D&D code generators create a ToolStrip control to capture and label these parameters.

  • Update management: Your UI needs some mechanism to refill once parameters change and when new data is to be added or changed data needs to be committed to the database. The D&D code generator adds buttons to the ToolStrip that point to your data management functions.


Using the BindingSource Class

You can spend considerable time trying to manage data binding if you stick with the 1.0 and 1.1 .NET Framework classes. One of the innovations introduced with the 2.0 Framework is the BindingSource class that you can instantiate in code, drag from the Toolbox, or get D&D to generate. This new class is especially useful when dealing with complex hierarchies of bound controlsfor example, when you're bound to Customers, Orders, and Items on the same form.

The BindingSource class acts as a proxy between a collection of bound controls and a specified data source. Basically, your code binds form controls to a data sourcejust as I discussed earlier with the control DataBindings propertyexcept backward. In this case, the binding is done from the top downnot from the bottom (the controls) up to the data sources. This means you can manage data binding from a single classassuming you can figure out how to program the BindingSource class. I'm going to try to help.

Tip

Behind the scenes, the BindingSource class creates and manages a list of type BindingList. I'll talk about that class a bit later.

Generally, a BindingSource serves several purposes by providing:

  • A layer of abstraction between the data and bound controls

  • A set of services used to navigate through the rowset, expose a "current" row, pass the current row data to bound controls, and pass data from bound controls back to the data source's rowset column values

  • A set of services to filter and sort the data returned from the data source rowset

  • A strongly typed data source to feed any control that can accept simple or complex DataSource binding

In earlier chapters, I saw the BindingSource class automatically included in my forms in a number of scenariosespecially as I toured the Data Source Configuration Wizard and used drag-and-drop to bind to a TableAdapter.

The BindingSource Properties

I'm not going to simply list the properties of the BindingSource, as you can get that information online or from the MSDN helpsuch as it is. I am going to point out properties (in Table 7.1) that you're likely to use when data binding in code and illuminate these. Many of these properties are used when writing tools that do data bindingtools like Visual Studio.

Table 7.1. Selected BindingSource Properties

Name

Description

Current

Returns a DataRowView that contains the current bound row

DataMember

Returns a string naming the DataTable referenced by the DataSource or an empty string when binding to a complex control

DataSource

Gets or sets the data source (as described in the text)

Filter

Gets or sets the expression used to filter which rows are viewed

Position

Gets or sets the index of the current item

Sort

Gets or sets the column name/order expressions

The Current Property

As with ADO classic Recordset objects, a bound rowset has a "current" record/row pointer. Yes, this pointer can be invalid at timesas when you haven't populated the rowset, you've executed Fill (or GetData) and there are no qualifying rows, or you've stepped off the end of the rowset. When binding to a populated DataTable, the Current property simply returns a DataRowView instance that contains the Row (usually a strongly typed row) addressed by the current row pointer. You manipulate this pointer by changing the Position property or by using the CurrencyManager class that I discuss a bit later. Figure 7.11 illustrates using the Current property.

Figure 7.11. Exposing the "current" bound row using the Current property.

[View full size image]

The Data Source Property

This property is one of the most important for the BindingSource instanceit points to the source of data. Typically, this will be an instance that supports IBindingList like the DataTable or the DataView. Figure 7.12 shows how I set the DataSource for a set of TextBox controls and for a complex bound controla DataGridView in this case. Note how the Timestamp column in the DataGridView is hidden so it does not throw DataError exceptions when it's rendered.

Figure 7.12. Setting the BindingSource DataSource property.

[View full size image]

One problem you might encounter when working with multi-column complex bound controls (like the DataGridView) is that you don't have any control over how the control initially displays the data. That is, when you bind to a complex control, it's an "all or nothing" contract. You bind to either all of the columns or none of themyou don't get to pick and choose. This is not the case when you use drag-and-drop to "details" (individual simple-bound controls).

In some cases, I've seen the DataGridView throw exceptions (firing the DataError event) on each row that contains a Timestamp byte arraybut only when the value is made visible. It seems that there is no way to display the binary data in a byte array. One approach to deal with this would be to hide that column. That's not hard to do when you drag and drop a TableAdapter to the Form, but when you roll the data binding yourself, things get a bit sticky. In this case, you won't have any schema to alter until the control is bound at runtime. If this column is initially visible, you're pooched. About the only thing you can do is make sure that columns containing unrenderable data types (like Timestamp) are not included in the query, or set the Visible property on the troublesome DataGridView Column to False, as shown in Figure 7.12.

No, you don't have to build a DataTable to display data in a bound control. As I said, you can bind to anything that supports one of the acceptable interfaces, like IBindingList. For example, suppose you want to display a pick list, such as a list of valid columns. In this case, you can construct an array list or Collection to hold the data, populate the Collection, and set the DataSource property to point to the collection. An example of this technique is shown in Figure 7.13.

Figure 7.13. Binding to a Collection.

Yes, when the contents of the colColumns Collection are changed, the binding mechanism posts the changes to the bound controlsjust as if they had come from an in-memory DataTable.

The Filter Property

The BindingSource Filter property is very much like the DataView Filter property, as it supports the same syntax. Basically, the Filter property works like a WHERE clause. It's used to filter the in-memory bound data source so that only rows that meet the criteria in the Filter string are made "visible" to the binding mechanism. This way you can temporarily hide rows that the user does not want to see. The Filter property is coded as <Column name> <expression>. All string literals are framed in single quotes. You can use AND or OR or other Boolean operators to combine the filtering criteria. For example, the following strings are valid Filter property settings:

  • "Author LIKE 'Va%'"

  • "Year_Born > 1940 AND Year_Born < 1948" (no, the BETWEEN operator is not supported)

  • "Author LIKE 'O''M%'". Note that the apostrophe is doubled to deal with the "O'Malley" issue.

  • "NOT Author IS NULL". This filters (out) rows where the Author column is NULL.

  • "Au_ID > 35 AND Author LIKE 'B%' OR Year_Born = 1947

Note the use of the SQL Server "%" wildcard character. All string literals are framed in single quotes (the apostrophe character,'), so you'll have to deal with the "O'Malley" issue when you collect filters from your UI.

As new rows are added to the DataTable bound to the BindingSource, if they don't meet the Filter criteria, they'll disappear once the add is committed.

Remember, the Filter property does not delete rows that don't meet the Filter criteriait simply hides them. This does not reduce the memory footprint your data consumes, and it does not reduce the amount of time it takes to get the data to your client or improve performanceunless you consider the round-trip overhead to fetch a filtered resultset. That is, it might make more sense to fetch fewer (filtered) rows in the first place instead of filtering a larger rowset on the client. However, if you've already reduced the number of rows to a scalable number, using the Filter property to further refine the bound rows can make a lot of sense.

Using the Position Property

The BindingSource Position property is used to determine or set the "current" row index. I discuss how to set and manage this property a bit latersee "Using the Move Methods."

Using the Sort Property

Again, this property is implemented like the ADO.NET DataView Sort property. This means you can name any column and indicate "ASC" (the default) for an ascending sort or "DESC" for a descending sort. Yes, you can sort on more than one column by simply separating individual sort expressions with commas. Actually, this property might not be needed, as complex bound controls like the DataGridView have sorting built in. It's a lot more interesting for ASP applications or in situations where you need a multi-column sort.

Of course, you can pre-sort the data being fetched from the server using an ORDER BY clause in the initial SELECT query. However, consider that SQL Server's query optimizer might be forced to sort millions of rows on the server before picking the few rows to return to your client. In this case, accepting an unsorted rowset and using the Sort property here (or on a DataView) makes more sense.

The BindingSource Events

As rows are added and changes are made to the bound controls, the BindingSource serves as a point source for events that can report these operations. It's well beyond the scope of this book to wade into best practice or implementation of these events, so I'll just touch on the high points.

In the example application used to illustrate these events, I drag the Toolbox BindingSource to the form. This makes it easy to set up the event handlers in the Visual Studio UI. Another approach (in Visual Basic) is to declare the BindingSource With Events to enable the event handlers in the Visual Studio UI. However, the Visual Studio UI can make this even easier by using a new feature that exposes events of any item you can select with the Design window openassuming the object is present on the form. That's why I dropped a BindingSource Toolbox icon on the form. To demonstrate this new feature, follow these steps:

1.
Open a Windows Forms project in either Visual Basic .NET or C#.

2.
Open the Toolbox and navigate to the "Data" tab. Drag and drop the BindingSource icon to your form (anywhere). Notice that a named BindingSource instance is created in the Tool Tray area.

3.
Click on this new BindingSource icon and press F4 to open the Properties page.

4.
Click on the yellow lightning bolt icon, shown in Figure 7.14. This enumerates the object's events and any event handlers that have been assigned. To create an event handler prototype in code and open this in the code editor, simply double-click on any of the exposed events.

Figure 7.14. Enumerating the event handlers for a selected object on the Form.

Table 7.2 lists and briefly describes the events exposed by the BindingSource class. I wrote an example that monitors each of these events (as shown in Figure 7.15). As you can tell by running this example yourself (it's on the DVD), there are quite a few events to monitor as rows are added to the binding list and as the current row is changed. Only the ListChanged event returns any additional information. It returns the type of change and a PropertyDescriptor that describes the change. The "list" referred to in the table is the rowset managed by the BindingSource.

Table 7.2. Selected BindingSource Events

Name

Description

AddingNew

Fires before an item is added to the underlying list

BindingComplete

Fires when all the clients have been bound to this BindingSource

CurrentChanged

Fires when the currently bound item changes

CurrentItemChanged

Fires when a property value of the Current property has changed

DataError

Fires when the bound control is incapable of dealing with the data for some reason

DataMemberChanged

Fires when the DataMember property value has changed

DataSourceChanged

Fires when the DataSource property value has changed

ListChanged

Fires when the underlying list changes or an item in the list changes

PositionChanged

Fires after the value of the Position property has changed

Figure 7.15. Example application tracks BindingSource events.

[View full size image]

The BindingSource Methods

The BindingSource has a rich complement of methods that help you manage its collection of data bindings. However, after consulting with Brian Noyes, I've discovered that many of these methods can't be used when binding to DataTable instances. The list in Table 7.3 shows the most interesting of the methods you can use along with a brief description. I'll further illuminate some of the less intuitive or more complex of these methods. I've also left off some methods that are implemented to help the inner workings of Visual Studio or its code generators, as they are far too complex for general use.

Table 7.3. Selected BindingSource Methods

Name

Description

CancelEdit

Cancels the current edit operation

EndEdit

Applies pending changes to the underlying data source

Find

Find the specified item in the data source

MoveFirst, MoveLast, MoveNext, MovePrevious

Changes the "current" row (Position) in the bound list

RemoveFilter, RemoveSort

Removes the filter or sort associated with the BindingSource

ResetBindings, ResetCurrentItem, ResetItem

Causes a control bound to the BindingSource to reread all (or the selected) items in the list and refresh their displayed values

ResumeBinding

Resumes data binding after suspending with SuspendBinding

SuspendBinding

Suspends data binding to prevent changes from updating the bound data source

Using the EndEdit and CancelEdit Methods

The EndEdit method is probably one of the most overlooked methods when it comes to data binding and updating. It's used to inform the databinding mechanism that the changes made to the bound controls are ready to be posted back to the bound control column(s). Wait a minute. Isn't that supposed to happen automatically with data bound values? Sure, you're right, it is. However, how is the binding mechanism supposed to know when you're done with your edits? Should the changes be posted as you type each letter or paste in a new value from the Clipboard? What about when you move the cursor, tab from column to column, or simply click on a new column in the same row? Does that mean you're done with the changes? What about when you click outside the scope of the bound control on another button, or outside the scope of the application on another application or on a system-level function? Actually, few, if any, of these events consistently trigger the operation that posts changes made to a control(s) back to the bound DataTable columns. It's for this reason that you need to remember to execute the EndEdit on the BindingSource to inform it that any changes need to be written to the DataTable. In Chapter 13, "Managing SQL Server CLR Executables," I code the EndEdit as part of an update routine. It doesn't really matter, as long as it's done before you call Update or test a DataTable for changes (with the HasChanges property).

Let's walk through a typical scenario that trips up a lot of developers. For example, when working with the DataGridView (or any grid control) and editing a cell in the grid, developers don't always realize that changes are being made to the gridnot to the underlying bound DataTable. If they happen to navigate to another grid row, this triggers the "end edit" functionalitythe cell contents are persisted to the DataTable. Typically, this works in testing, but when actual users work with your application, they might simply enter the data into cellsassuming that the changes are being posted to the DataTable (and the database). Sometimes they are, but sometimes they're notdepending on the sequence of events. It's for this reason that you need to ensure that before focus leaves the DataGridView, you fire the EndEdit event or provide a "Save" button on the form to execute this method. No, this does not write anything to the database. Your code still needs to execute the DataAdapter or TableAdapter Update method to post changes to the database.

The CancelEdit method simply tells ADO.NET that any changes not already posted to the DataTable are to be ignored. That is, if a bound control's contents are refreshed (as when you execute the ResetBindings method), these changes are lost.

Using the Find Method

The BindingSource Find method is fairly intuitive and brutally simple. It searches the bound column specified and returns the row index of the first row that exactly matches the search argument. No, it does not support ambiguous search, and, no, it does not permit you to set the starting row (which limits the usability of this method). The snippet shown in Figure 7.16 illustrates how to call the method. I use the integer returned from the method to change the DataGridView currently selected row indexassuming it points to the underlying data row I found.

Figure 7.16. The Find method returns the first row index that matches.

[View full size image]

This method won't be that useful if you need to search for ambiguous values or repeat the search when there are two or more matches in the bound column. In this case, I recommend binding to a DataView class exposed by the DataTable. It has a far richer set of search methods and properties. I discussed these in Chapter 6, "Building Data Sources, DataSets, and TableAdapters."

Using the Move Methods

Reminiscent of the ADO classic Recordset object, the BindingSource "move" methods (MoveFirst, MoveNext, MovePrevious, MoveLast) are used to reposition the "current" bound row index when using simple bound controls. This index is exposed in the Position property, which can be used to set a specific row index. The move methods are quite intuitive to code and deal automatically with "end of file" and "beginning of file" (EOF/BOF) issues. That is, no exception is thrown if you try to set the current row off the end of the bound list. This approach is especially useful when simple binding to "display" controlsindividual simple bound controls instead of a complex bound control like the DataGridView.

To demonstrate these methods, the example application includes a custom ToolStrip control I created and a UI that has buttons for each method. Nope, there is no other exception handling or other code that the ADO classic Recordset navigation code requiredit's really simple to implement. Incidentally, the populated ToolStrip, shown in Figure 7.17, can be automatically generated if you use drag-and-drop binding. It's also an ideal candidate for use as a compiled custom control.

Figure 7.17. The ToolStrip control is used to host other controls.

As I said, the Position property reflects the current row index. It's a zero-based index (as is the Row collection index). If it's set to an out of range value, the .NET Framework sets it within the valid range of bound rows0 to the number of rows in the bound DataTable. All you need to do is verify that the value passed is numeric, as shown in the event handler in Figure 7.18.

Figure 7.18. Managing the Position property.

Data Binding Using the BindingSource Class

At this point, I think that we're all ready to walk through some typical data binding scenarios. As I see it, there are several segments to this discussionassuming you've already executed a query and your data is ready to display and (possibly) edit:

  • Displaying data in "complex" bound controls: This is really pretty easy, as the control itself (like the DataGridView) knows how to handle much of the overhead involved in rowset population, current row navigation, data validation, and other issues.

  • Displaying data in "simple" bound controls: In this case, you have a bit more work to do. First, you have to expose a separate control for each column of the data structure to display. Each control (like a TextBox, Label, or ListBox control) must be added to the list of controls in the BindingSource. You'll also want to implement some type of "cursor" (current row) positioning mechanism. That's what the CurrencyManager is forI'll talk about that later.

  • When the data is changed: You have more work to do. What most of the Microsoft help topics don't discuss is how to perform UI data validation. While I didn't include a chapter on that subject, in essence, this is the process of testing the values entered to ensure that they conform to specific business rule criteria. I hope to write an EBook to tour my ideas on UI design and implementation. These rules can be implemented in business classes, by morphing the bound control properties to match the rules, or in a number of other ways that I discuss later.

  • When data is removed or added: The bound controls need to be made aware of these changes. I'll talk about how to refresh simple and complex bound controls later in this chapter.

  • Moving data back to the local data source: Once the data is committed, some process needs to move the data from the bound control back to the data source. This seemingly simple step is often overlooked by developers who assume that because the data is bound, the Framework somehow magically moves the data after it's changedit doesn't. I'll discuss later how to know when the data has changed and how to force the Framework classes to write it back to the data source.

  • Writing data back to the database: Once the changes are written back to the data source, you'll need to somehow get it written back to the database, if you expect it to be there the next time someone queries the data. This is the subject of yet another chapter that discusses how to write updated data back to SQL Server. See Chapter 13 for more details.


Using the BindingNavigator Control

The BindingNavigator control is added to your form when you use drag-and-drop binding, but you're free to use it on your own to build a custom data binding mechanism. This control inserts yet another layer between your user and the BindingSource. In my example, I used the BindingSource control dragged from the Toolbox to wire to a strongly typed Data Source. Can it be used with a conventional DataTable (or, better yet, its DataView)? Sure.

By default, the BindingNavigator control is hard-docked at the top of your form, but you can set its docking location to other locations on the form. Its default configuration (as shown in Figure 7.19) includes most of the same buttons I used in my custom ToolStrip control (shown previously in Figure 7.17), along with a couple others. It's a handy way to get a running start on your own custom ToolStrip control.

Figure 7.19. The default layout of the BindingNavigator control.

Note that while the default BindingNavigator button configuration includes "+" (AddNew) and "X" (Delete) buttons, it does not include a "Save" (Update) button for some reason. However, if you click the "tasks" button, you can generate an entire series of "standard" buttons, including the save, print, help, and other function buttons shown in Figure 7.20. On the other hand, this bevy of beautiful buttons may be too bountiful a blessingforcing you to implement more code than you had planned for. If you use drag-and-drop data binding, the Visual Studio UI adds the Update (diskette) icon to the BindingNavigator ToolStrip for you.

Figure 7.20. The addition of "Standard Items" to the BindingNavigator ToolStrip.

The BindingNavigator control's property page exposes a number of canned methods to help set up calls to appropriate code to execute the move methods discussed earlier in this chapter, as well as to the data action query invocations to add a new row and change or delete existing rows. I suggest you use drag-and-drop binding to see how Visual Studio configures the control and connects it to the BindingSource properties and methods.

Once you have a bindable data source, setting up the BindingSource and BindingNavigator controls is a bit tedious but not that hard. What's missing are a few nuances that the code generated by drag-and-drop takes care of for you automatically. For example, once you set all of the method pointers in the BindingNavigator properties window (as shown in Figure 7.21), you'll still need to set the BindingNavigator CountItem and PositionItem properties. That code, along with the other hookup code, is included in the BindingNavigatorExample on the DVD.

Figure 7.21. Set the BindingNavigator Move method pointers.

Previous Page
Next Page


Using the DataGridView Control

As I said .earlier in this chapter, the DataGridView control is the new replacement for all of its earlier successors exposed one at a time over the decade since the introduction of Visual Basic. No, I don't really think it was written by a summer intern or new employeeit's really pretty sophisticated. It makes a good way to expose rows from a DataTable or DataView instance. Once it's populated with schema, it's easy to customize. However, you can't pre-populate the control until you have a data source with which to populate it. Yes, by far the easiest way to bind to data is by dragging a TableAdapter to the Form, which gets Visual Studio to instantiate a DataGridView and set its properties to map the DataTable columns. Once you've done so, you can click the "tasks" button on the upper-right corner of the DataGridView to expose the "DataGridView Tasks" menu, shown in Figure 7.22.

Figure 7.22. Setting options with the tasks dialog.

[View full size image]

The tasks dialog permits you to determine how the DataGridView is to behavewhether or not it supports updates or column reordering. It also permits you to set docking behavior as well as how the columns are displayed. You can also add columns that must be custom bound.

Clicking on the Edit Columns... link opens the Edit Columns dialog, shown in Figure 7.23. This gives you an opportunity to set properties like ReadOnly (to prevent changes to the column) and the ability to sort the column through the DataGridView UI. Note that the Au_ID column (the primary key) ReadOnly property is automatically set to True.

Figure 7.23. Editing columns configured in the DataGridView.

[View full size image]

For read/write columns that are bound to string values, you can set the MaxInputLength to prevent strings that are too long. Unfortunately, this value is not automatically set, so if the user enters a string longer than that defined in the underlying DataTable, your code will throw an exception. I suggest adding code to your application to automatically set the MaxInputLength property based on the defined column width. The DataGridView also does not expose the datatype of the underlying DataTable Column, but the DataGridView knows what it is and makes sure that any value you enter conforms to the type's formatting and range restrictions. For example, if you bind to a date, any value entered into the DataGridView column must be recognized as a valid date value. The same criterion is applied to numeric types, including range-checking integers, currency, and other numerics. Be prepared to trap the DataError event when these validation rules are violated. I discuss this issue again in Chapter 13 when we visit the update operations.

What happened to "Auto Format"?

During the beta tests, the "Action" dialog (which the "Tasks" dialog replaced) included an "Auto Format" option that permitted you to choose a decorative skin for your DataGridView control. While it was a popular feature of the old DataGrid control, it's not available (yet) for the DataGridView.


Using the ProgressBar Control

As I mention elsewhere in the book (any number of times), it's not how fast your application runs that makes a user happy, but how fast it seems to run. The ProgressBar and other real-time feedback controls can help you build the appearance of good performance into your application. As I'll discuss later in the sections on handling ADO.NET asynchronous operations, you can simply use the UI thread to increment a ProgressBar controlbut only if it's not busy running an ADO.NET method or other blocking operation.

Actually, if all you want is an animated control and you don't care if it actually reflects the real "progress" of an operation, then simply set the ProgressBar control's Style property to "Marquee", as shown in Figure 7.24. In this case, the moving bars simply move from one end of the ProgressBar to the other and repeat. I'll let you experiment with the numerous mechanisms to increment the other styles.

Figure 7.24. The three ProgressBar Style property settings in action.

If the operation you're executing while entertaining the user with the ProgressBar is going to take more than 10 seconds or so, you should consider how to let the user know how much time the operation is really going to take. Unfortunately, as you well know, execution time is not always easy to determine in advance. It's like trying to predict when the ocean levels will reach the steps of the Capitol building. The length in time of computer operations is determined by the performance of the system, the system workload at the time, and, when working with SQL Server, how much traffic is on the Net, the server workload, and a dozen other factors outside of your controllike having police setting up barricades around your house. Your task might not ever finish due to one of these unforeseen factors or simply because you executed a query that can't be completed before Microsoft stops supporting your version of Visual Basic.

When I want to keep the user from rebooting the system while a long operation is executing, I take one of the following approaches:

  • "Fake" the progress. That is, I start a progress bar and set the upper limit to a very high number. I then increment the value every few seconds and increment the value halfway up the scale. That way (in theory), the progress bar never really reaches the end. What it does do (which is bad) is appear to stall several seconds into the run. This is what Access did at one timebut it used a very long progress bar. This approach still causes users to close the application after awhile, as they think it's stuck. That's exactly opposite of what we want.

  • "Guess" the endpoint. When I run any query, I have a rough idea of how long it's going to take. This value is based on history, experience, conditions I do control, and other deterministic factors. I then set this value plus a fudge factor as the Maximum property setting in the ProgressBar control. If I do reach this Maximum value, I know (suspect) something went wrong, and I give the user the opportunity to bail or keep waiting. Perhaps the user knows something I don't. The next time I run the same routine, I know that in the last run, it took quite a bit longer (or shorter), so I set the Maximum property to reflect current reality.

  • "Never stop". An approach used by Microsoft Office is to slowly increment the ProgressBar but, when it reaches the Maximum value, restart it. This is not much better than the Marquee style IMHO.


Summary

Yes, there are a lot of other controls available to Windows Forms data access applications, and even more available to ASP.NET applications. However, I simply don't have the bandwidth to document, illustrate, and explain each and every one of these. There are other resources out there, so be sure to consult my "Suggested Reading" list for more details. The Visual Studio tools suite keeps expanding, and, for the most part, this simply makes it easier for developers to provide richer UI without adding custom controls and a lot of code to support them. That's the whole idea behind controlsthey encapsulate commonly used operations into black-box or generated code blocks incorporated into your application. Thankfully, Visual Studio is generating more code and depending on fewer black boxes to make developers jobs easier.


Chapter 8. Getting Started with ADO.NET

    Introduction

    Approaching Data Access Challenges

    ADO.NET from 50,000 Feet

    Using the Visual Studio Object Browser to Explore ADO.NET

    Instantiating ADO.NET Objects

    Exploring the System.Data.SqlClient Namespace

    Exploring the SqlClient Namespace

    Exploring the System.Data Namespace

    Summary


Introduction

This chapter brings you up to speed on the ADO.NET classesthe foundations and structures upon which all ADO.NET applications are built. In subsequent chapters, I'll apply these ADO.NET classes to a wide variety of tasks you'll use in your applications. I suggest touring the wide streets of the ADO.NET object model and exploring the darker streets and alleys of the underlying base classes from there. To this end, I discuss the ADO.NET namespace in several sections to help you better understand which parts of ADO.NET leverage the others. I'll also briefly examine the underlying Framework base classes. The classes you code against inherit from base classes, so it's interesting (but not critical) to know what functionality they implement. No, you don't really need to know that the SqlConnection class inherits from DbConnection class that inherits from IDbConnection, but for those of you that like to know how ADO.NET is implemented behind the scenes, I'll provide a pointer or two. Don't be put off by the number of examples in this chapterthis is where I roadmap later chapters to let you explore implementation details and examples later on.

Over the last eight years or so, I've written four books and countless articles on ADO.NET. I struggled with the question of cutting and pasting from those books into these pages, but I decided to simply update you on the most recent changes, highlight the Visual StudiotoADO.NET code generators, and point you toward my old books and several others whose authors I respect. Sure, I've added quite a bit of new content here and several new examples. Frankly, there are really too many books out there on ADO that say about the same thing. That's the publisher's fault. I'm not going to wade into that street fight swinging. Instead, I'll sit back, pick a couple of the winners, and send you to them. Yes, my ADO and ADO.NET Examples and Best Practices (Apress, 2002) is dated, but you'll be able to find it in used bookstores. Be sure to see my Suggested Reading List for more titles.

If you really want to know how ADO.NET does what it does behind the scenes, I suggest using one of the publicly available class browsers such as Lutz Roeder's Reflector[1]. These tools reverse-engineer the source code that's used to implement Framework classes. I've made the developers at Microsoft a bit crazy by telling them how to fix bugsreferencing the source code in the framework when I do so.

[1] See www.aisto.com/roeder/dotnet/


Approaching Data Access Challenges

Before I wade into ADO.NET, we need to pause briefly and talk about an alternativethe Data Access Application Block (DAAB). For some reason, Microsoft felt that developers needed another way to create data access applications without ADO.NET. I expect they wanted to further distance themselves from a "low-level" data access paradigm that could be used with any upcoming data access interface. To this end, they created the DAAB sample data layer to make the process of working with data "easier". As I see it, the DAAB won't help much, and it could just slow you down. For one thing, it requires that you learn an entirely new set of classes, methods, and properties, as well as a new set of behaviors, constraints, and limitations. New people coming into an organization who are required to use the DAAB will have to learn your team's coding regimen and the DAABeven when they know how to code ADO.NET with acuity. Upon closer inspection, I (and many others) have found the DAAB makes some of the same mistakes made by other applications using ADO.NET directlybut without the benefit of being able to easily trace through "straight" ADO.NET code. I've heard reports of leaking connections and other issues that can be eliminated through "best practices" coding techniques. As the 2.0 Framework ships, there is no 2.0 version of the DAAB, but I expect it to catch up eventually. Whether or not you should use it is another question.

Developers have to make some fundamental decisions before writing their first line of data access codeincluding whether or not to use the DAAB. Several of these decisions have to do with the balance between client-side ADO.NET code and server-side T-SQL (and CLR) code. Should ADO.NET be tasked with fetching and updating database data, or just fetching and executing custom update logic that posts changes as parameters back to the database to be posted to the database via stored procedures? For example, in order to use the SqlDataAdapter Update method to post changes to the server, developers have to plan a data access strategy that returns rowsets from a single table and manage the DataTable objects individually. This usually means you'll have to create and configure "N" SqlDataAdapter objectsone for each database table you intend to update.

The 2.0 Framework and Visual Studio have attempted to make this process easier (in a way) by introducing the strongly typed TableAdapter class, which can make this table-centric approach easier. While the jury is still out on the TableAdapter, many developers have abandoned the DataAdapter Update approach (by whatever means) in favor of calling SQL Server stored procedures to post changes to the database. A big reason behind this decision is that their protected SQL Server databases are procedure-driven. That is, the DBAs have already locked out access to base tables, and all updates must be done via stored procedures which write to one or many tables using server-side transaction management. I suggest you consult with your DBA before launching out on any far-reaching data access strategyit will save you time and hair follicles in the long run.

At least one of my reviewers chides me on my steadfast endorsement of designs built on stored procedures, and some at Microsoft and others in the industry agree. They would have you build applications using Object Relational Mapping tools and designs that (IMHO) put you even further behind the learning curve. They would have you manage transactions at the client instead of in a common procedure that all applications can leverage. I expect that the well-worn and well-understood stored procedure designs will be around for quite some time, despite those that say we're just being hard-headed about change. Perhaps we are, but perhaps we like to stick with architectures that simply work.


ADO.NET from 50,000 Feet

Let's start with a high-level view of ADO.NET to get an overall idea of how it works and why it's needed. I've found that ADO.NET is simpler to understand (and explain) if it's broken down into two parts:

  • The classes used to interact with the data source: These classes are implemented by provider-specific code designed to connect to the data source, collect and pass the queries to the SQL engine for processing, and return the resultsets, rowsets, and exceptions.

  • The classes used to manage data in memory: These classes are populated by the provider-specific classes and basically implement simple in-memory data structures that support sorting, filtering, and finding, as well as the binding interfaces and classes used to connect related rowsets.

You might have heard that ADO.NET implements a disconnected in-memory "database." That's not really true. It's really a stretch to call the classes that ADO.NET exposes a "database" in the sense of a "real" database management system like SQL Server, SSEv, or other DBMS engines like Oracle or DB2. Data access interfaces like ADO.NET (and its predecessors) do not implement a JOIN or SELECT engine driven by SQL or even methods and properties. All data access interfaces simply connect to DBMS engines or other data sources that do the heavy lifting.

While ADO.NET was designed specifically to deal with disconnected data structures that ASP.NET applications require, these same ADO.NET classes can be used to manage "connected" data as wellat least, to an extent. No, ADO.NET does not know how to build or maintain any form of server-side cursoryou'll have to implement that yourself or solve your problem using a disconnected architecture. Perhaps, Appendix IV, "Creating and Managing Server-Side Cursors," will helpit contains a whitepaper I wrote a while back that shows you how it's done. Connected applications form the core of many sophisticated applications, and while ADO.NET is not as rich as ADO classic in its ability to support this type of application, many applications are still being built and converted to access data using connected strategies.

IMHO

I don't really expect Microsoft to ever implement a server-side cursor for ADO.NETthey've had over seven years to do it, and they have not made any forward progress toward that goal. They're too focused on re-creating ADO.NET again....

Working with Data Sources

As I said in earlier chapters, you don't need to access a DBMS to gather data and manage it with ADO.NET. Data can be extracted from anywhere it's storedfrom memory, to flat files, to sophisticated relational database management systems like SQL Server, to simple DBMS systems like SQL Server Everywhere. However, unless you're making up data out of thin air (as some politicians seem to do) in order to fetch your data, you'll probably want to use an existing .NET Data Provider (as I discuss later in this chapter) or create your own. While it's not hard to write and expose a custom data source, you won't have to go to the trouble for virtually any kind of data because there are existing .NET Data Providers for almost every conceivable data source, as well as one-size-fits-all (OSFA) data providers that permit you to use existing ODBC and OLE DB drivers and providers to access legacy data.

Focusing on SqlClient

For most of this book, I'll be focusing on the SqlClient .NET Data Provider, as it's specifically designed to access Microsoft SQL Server Version 7.0 and later. For earlier versions of SQL Server, I recommend the System.Data.Odbc .NET Data Providertry to avoid use of the OleDb namespace, if possibleunless you don't care about performance. If you have to use one of the other providers, some of the rules change. For example, ODBC and OLE DB use different techniques to mark parameters in T-SQL queriesthey don't support named parameters. For best results, see my earlier books that discuss these older technologies.

Essentially, the System.Data.SqlClient namespace contains classes used to create objects used by the System.Data namespace. No matter what data source you use, all roads lead to the System.Data namespace, where the DataTable, DataSet, and other essential data manipulation classes live. To illustrate this, Figure 8.1 shows a simple application's use of ADO.NET. Here I open a SqlConnection and use a parameter-driven SqlCommand to return a SqlDataReader. This data stream is used to populate a DataTable bound to a DataGridView control and pass back changes via a SqlParameter object and another SqlCommand.

Figure 8.1. ADO.NET basics.

Previous Page
Next Page


Using the Visual Studio Object Browser to Explore ADO.NET

To get a better understanding of the ADO.NET namespaces, I suggest that you open Visual Studio and the Object Browser (View | Object Browser), where you'll see a window similar to that shown in Figure 8.2. By using the OB, you can get a better idea of how to address a specific Framework class, get (limited) help on selected objects (via F1), and view the properties, functions, methods, and events of the objects. Don't be alarmed at the sheer volume of classes in the Framework. While I've never counted them, I understand there are over 70,000 of these classes, depending on what's included in the total. Sure, you can gain a complete understanding of ADO.NET development without memorizing these namespacesbut there will be a test on them later.

Figure 8.2. The Visual Studio 2005 Object Browser exploring the Framework Namespaces.

Since this book focuses on SQL Server, I spend considerable time discussing the SqlClient .NET data provider and far less time discussing the OleDb or Odbc providers. Frankly, my previous books on ADO.NET[2] provide quite a bit of detail on these other providers and how to connect to their data sourcesI won't be revisiting that material here. Once connected, the functionality between providers is very similar, so the concepts you learn when working with SqlClient are similar to those applied to OleDb and Odbc.

[2] ADO and ADO.NET Examples and Best Practices for Visual Basic Programmers (Apress, 2002) and ADO.NET Examples and Best Practices for C# Programmers (Apress, 2002).

Exploring ADO.NET with a Class Diagram

Visual Studio 2005 includes a new tool called the "Class Diagram" that shows the interrelationships between the base and working classes I code against. If your application builds a number of classes (especially when you're coding complex classes), the Class Diagram tool can make it easier to visualize the classes and their interrelationships, and actually generate your own classes from scratch. If you don't use it for anything else, a Class Diagram can be an effective way to document your classes. I won't have time to teach you how to use the Class Diagram, but I do want to touch on it briefly to give you an idea of its power. To that end, let's create a simple Class Diagram using the Object Browser and a new Class Diagram tool.

1.
Create a new Visual Studio 2005 Windows Application project. Sure, any type of project can contain a Class Diagram. The example creates a Windows Smart Client from the HHGTemplate, but you can use any template you choose. This template is provided with the other examples on the DVD. It simply starts your application off with a user-defined set of references, classes, and commonly used objects.

2.
Right-click on the Project, choose "Add New Item", and choose "Class Diagram" from the "Add New Item" dialogname it "cdExample.cd". This adds a new file to your solution. Visual Studio displays the new Class Diagram design surface in a new windowit instructs you to drag items from the toolbox and visualize existing classes by dragging items from the Class View or Solution Explorer. What it does not tell you is that you can drag classes from the Object Browser as wellthat's what I'm about to do.

3.
Click on the View | Object Browser menu to expose the Object Browser.

4.
Right-click on the "cdExample.cd" tab and choose "Create new vertical tab group"I want to be able to see both the Object Browser and the Class Diagram at the same time. With the Object Browser and Class Diagram side by side, I can easily drag classes from the Object Browser.

5.
Scroll down in the Object Browser until the System.Data.SqlClient namespace is exposed. You should see the same diagram shown in Figure 8.2. Click on the "+" sign to expand the namespace.

6.
Click on the SqlConnection class in the Object Browser. This exposes the list of classes, properties, methods, and events associated with the SqlConnection class.

7.
Right-click on the right pane of the Object Browser and choose "Group by Member Type". This makes it easier to find each type of class within the selected namespace.

8.
Drag the SqlConnection class to the Class Diagram. Left-click on the "SqlConnection" icon in the Object Browser and drag it to the Class Diagram pane. You should see a graphical representation of the SqlConnection class in the Class Diagram, as shown in Figure 8.3.

Figure 8.3. Using a Class Diagram to explore the SqlConnection class.

[View full size image]

9.
Right-click the "SqlConnection" graphic in the Class Diagram and choose "Show Base Class". This creates a graphical representation of the DbConnection class. If you expose the base class of the DbConnection class, you'll find that it's inherited from the Component class (like all Framework classes).

10.
Right-click on each class you've exposed and choose "Expand" to view the Properties, Methods, and Events implemented by each class. Your Class Diagram should look like Figure 8.4.

Figure 8.4. The Visual Studio Class Diagram can expose class properties, methods, and events.

[View full size image]

Okay, what can be learned from the diagram in Figure 8.4? Well, notice that the DbConnection class implements several properties and methods that are common to all ADO.NET "connection" classesregardless of the data access provider. Other properties are exposed only on the provider-specific class. For example, SqlConnection implements the PacketSize and StatisticsEnabled properties and the ClearPools method.

The Class Diagram can also be used to show property associations between classes. Let's add a SqlCommand class to the diagram and illustrate this point.

1.
Right-click each class in the Class Diagram and choose "Collapse".

2.
Drag the SqlCommand class from the Object Browser to your Class Diagram. Note that you can reposition any of the graphical representations on the Class Diagram to make them easier to view.

3.
Expand the properties on the SqlCommand class graphic, and right-click the "DbConnection" property. Choose "Show as Association". This connects the SqlCommand class graphic with the graphic for the class that implements DbConnection. Your diagram should look like Figure 8.5 at this pointchoose "Adjust shape width" to expand the property and method names.

Figure 8.5. The Class Diagram can be used to show Property/Class associations.

[View full size image]

As you can see, a Class Diagram can be a helpful tool to visualize how the ADO.NET classes inherit from one another. This same technique can be used to visualize your own custom classes. Consider that the Class Diagram tool can be used to generate classes as well. I'll let you explore those features on your own. Be back here before the bus leaves for the next stop.

Deciphering the Namespaces

One of the problems you might face when trying to understand ADO.NET seems to be exacerbated by the Object Browserit does not seem to show the "real" object names for many of the objects you're likely to access in code. For example, to address the collection of DataTable objects managed by a DataSet object, you code something like the following Visual Basic .NET code (shown in Figure 8.6) that references the DataSet Tables property.

Figure 8.6. Addressing objects in the ADO.NET namespace is simpler than it seems.

[View full size image]

IMHO

I like to hide (for the most part) the underlying .NET Framework classes in the examples I provideI think it makes the code easier to read.

If you have a compelling reason to dig deeper into the SqlClient (or any .NET Framework DLL), you can use Reflector to expose the underlying .NET Framework implementation. While writing this book, I've found it handy to use Reflector to debug more complex .NET issues by studying what ADO.NET is doing when its objects are referenced in code.

Getting a Handle on the Class Names

So, if you started your introduction to ADO.NET by simply drilling into the Object Browser, you might think that you would have to somehow code a reference to the DataTableCollection (as shown in Figure 8.7) when you want to refer to the DataTable objects exposed by a DataSet object.

Figure 8.7. Using the Object Browser to explore the System.Data namespace.

Fortunately, that's not the case, as you'll see when we tour the .NET Framework namespaces. While the DataTableCollection is a class, you don't refer to this class directly. Instead, it's referred to as the DataSet "Tables" property (which is of type DataTable), which points to a specific DataTableCollection instance. Get it? Well, if this is not yet clear, don't panic and sell your Microsoft stock. Visual Studio's Intellisense makes addressing the right object or collection of objects pretty easy. I'll revisit this issue again later in this chapter.

Using Shorthand to Address Classes

Those not familiar with .NET coding might wonder about the first line of Figure 8.8the Visual Basic .NET Imports statement (or the C# using statement). This notation tells the compiler to include this namespace when resolving class names. It does not change the application in any wayand, no, I don't know why these two languages use different terms for the same thing, since they were both created at the same time by (almost) the same company. I use the Imports or using statements to make coding easierat least, to write. Some feel that you should not use the Imports or using statements, as they hide the objects being referenced. This means when you see a class name in code, you won't know the namespace from which it's derived. For example, Figure 8.8 shows how one can reference a SqlConnection class in one of two wayseither directly or indirectly (depending on the Imports statement). You can have as many Imports (or using) statements as needed in your application. Note that the Using statement in Visual Basic .NET means something else entirelyin C#, it serves a dual purpose, but I'll get to those nuances later.

Figure 8.8. Using Imports to specify the namespaces to search when resolving references.

The Visual Studio Object Browser can be a handy tool when you want to explore the classes in selected namespaces or simply locate the namespace containing a class with questionable parentage. For example, if you find an example that contains a class name you can't resolve, you can use the Object Browser to search for all references to the classeven classes you define in your current project. Once you drill down into a specific class and view its functions, methods, properties, and events, you'll see a brief description of the object. In addition, you can press F1 to get object-specific help for the selected item. Of course, that assumes that the help engine is correctly wiredeven after Visual Studio 2005 shipped, the right (or meaningful) help topics don't always materialize when pressing F1.


Instantiating ADO.NET Objects

When it comes time to build your ADO.NET application, you'll spend considerable time constructing instances of the objects, populating their properties, and coding the method arguments. Again, Intellisense can help quite a bit here, but let's go over a few guidelines to help you get through the initial learning phase.

Understanding Code Scope

First, you'll need to choose where to instantiate your ADO.NET objects. For example, if you want the object visible to all Windows Forms in the project or all ASP pages, you'll need to create the object in a global "scope". The scope of an object is the region in which it is visible (addressable) in code. For example, in Figure 8.9, I've defined several objects that are visible in selected parts of the application. Note those objects created within a scope boundary are not visible outside that boundary. For Visual Basic 6.0 developers, this is a radical change, as the object scope rules have changed quite a bit from what they are accustomed to using.

Figure 8.9. Object scope determines where an object is visible.

[View full size image]

In Visual Basic .NET, scope boundaries include (but are not limited to) the Module, Class (in this case, the Form1 class), Function or Sub declaration, the TRy/Catch block, and If, Case, For/Next, Do/While, and other code blocks. In C#, it's a bit simplerobjects created inside curly braces ({}) are not visible outside the braces.

In the example shown in Figure 8.9, I create myTRyObject within a TRy/Catch block. This means the object will not be visible in the Finally/End Try block or outside the ShowScope Subroutine. On the other hand, the mySubObject is visible to all code blocks between the Sub and End Sub scope delimiters, but not outside of the Sub. The myGlobalObject SqlConnection object is visible to any routine in the form, but not outside the form. Likewise, if the application includes a Module, any variables declared as Public would be visible from anywhere in the application.

No, it's not a good idea to place your objects just anywhere. As I've discussed before and I'll discuss again, object placement is critical to scalability and the ability of the Framework to correctly construct and tear down the objects when they are no longer needed. For example, while it's possible to create an application-global SqlConnection object, it's not a great idea (as I'll illuminate in Chapter 9, "Getting Connected"). In ASP applications, it's usually best to create the SqlConnection just-in-time, so as to best leverage the power of the connection pool. Just remember, you can't depend on the .NET garbage collector to tear down and dispose of your orphaned objects, as you could with Visual Basic 6.0. Again, the concept of an application-global SqlConnection is generally not a good practice unless your application is very simpleand not an ASP.NET design. Another nuance I'll discuss in Chapter 9 concerns closing the SqlConnection object. Unlike ADOc, you should not depend on the .NET garbage collector to tear down objects as they fall from scope. Sure, the GC eventually disposes trashed objects, but by the time it does, your connection pool might be overflowing on the lawn.

Tip

Don't depend on the .NET garbage collector to tear down and dispose of orphaned SqlConnection or any other instantiated objects.

Understanding Object Instantiation

Once it's decided where the object is to be created (the breadth of its visibility), you need to decide how you want to instantiate it. Generally, I like to position an object in the correct scope but delay instantiating the object until just before it's needed. Instantiating an object simply means creating an "instance" or clone of the object's class. Think of it this way: A class is kinda like a cookie cutteryou know, those little trees, houses, and gingerbread men (people?) you used to cut out of sugar-cookie dough during the holidays. Each time you use the cutter to make a tree, you're instantiating an instance of the tree "class". When a class is defined, one or more New methods are included in the class code. These are used to generate and initialize the new instance of the classthis becomes your new ADO.NET "object". I could go into more detail about how the object properties are like candy sprinkles, but this is making me too hungry.

The New method (new in C#) is called a "constructor," and in the case of ADO.NET classes, it usually exposes a number of arguments used to set the properties of the newly born object instance. For example, when you want to create a new SqlConnection object, you can use one of two constructors exposed by the SqlConnection class. As you type the open parenthesis (as shown in Figure 8.13), Visual Studio's Intellisense kicks in and prompts for any arguments exposed by the implemented constructors (as shown in Figure 8.10).

Figure 8.10. Using Intellisense to choose a constructor.

Tip

Unlike Visual Basic 6.0, where I encourage you to avoid use of "Dim X as New Y" syntax, in the Framework, the New constructor is your friend.

In this case, the only constructors exposed by the SqlConnection class create the object with no arguments (option 1 of 2) or pass in a properly formatted ConnectionString as a string. If you choose option 2, the value supplied to the constructor as an argument is used to set the new object's ConnectionString property. Actually, the constructor you use is chosen automatically for you. Visual Studio matches up the values you supply by type to the most appropriate constructor.

You'll find that most ADO.NET objects have constructors to help you initialize (construct) the object without having to make unnecessary property settings. Those properties not provided by the constructor must be set manually. It's not unusual to code a New constructor and have to immediately add initialization code to set additional properties not exposed by any of the constructors.

Another nuance you might not be prepared for is the fact that the New constructor can throw an exception if its code determines that one or more of the arguments passed are incorrect. In the example, if the key/values pairs supplied at runtime in the ConnectionString string are invalid, the constructor throws an exception. This means you can place these object instantiation statements almost anywherein or outside the code path. However, if you expect that an exception can be thrown, the object should be instantiated inside a TRy/Catch block to prevent an unhandled exception from bringing down your application. This issue leads us back to the quandary of where you can (or should) scope the object. Again, this is another reason to set the scope of the object by properly positioning the object declaration in the right scope and only using the "as New" constructor in a scope that can trap an exception at runtime.

The Using Block in Visual Basic.NET

An alternative to coding the Dim statement in Visual Basic .NET has just been introduced with the 2.0 Framework. Visual Basic .NET 2005 now supports the Using block, as supported in C# from the beginning. The Using block can make sense when dealing with "unmanaged" resources, such as a file handle or a SqlConnection object. It's not necessary to code a Using block for managed resources, as the .NET Framework garbage collector (GC) handles disposal of these objects on its ownat least, eventually. The SqlConnection can be a good candidate for the Using block when its associated connection might be orphaned and left to rot in the connection pool while waiting for the GC disposal.

The beauty (and beast) of the Using block is that it not only defines the scope for an object, but it ensures that the object's Dispose method is called when the logic reaches the End Using or you branch out of the Using block. This can make it easier (and safer) to instantiate objects that must be torn down before leaving the current code scope. The problem is, if you plan to reuse the SqlConnection object later in your application, you might not want to execute its Dispose methodjust the Close method. Remember, Dispose flushes the state of all of an object's properties and releases the object's resources to the GC.

Figure 8.11 illustrates the Using block. As I get into the later chapters, I'll show plenty of examples where it makes sense to instantiate objects with a Using blockand even more where it does not.

Figure 8.11. Coding a Using block to manage a SqlConnection object.

[View full size image]

Notice that the example code passes the SqlConnection object to another routine, but in this case, the DoSomething routine is called, so the logic must return to ShowUsing, where the object is closed and disposed. However, if you try to branch out of the Using block as shown in Figure 8.12, the object is closed and the CLR disposes of the SqlConnection object as it should be. That's good because this eliminates the chance that the SqlConnection object would be orphaned, but it's bad in the sense that the routine that called ShowImproperUsing won't get a valid object.

Figure 8.12. Improper use of the Using block.

I suspect that your solution to this Using block problem would be to eliminate the Using block. This would be a mistake because it opens up an opportunity to orphan the connection object. An "orphaned" object is one that can no longer be addressed and is waiting for the GC disposal. When this object is a SqlConnection object, it also means a connection is orphaned in the connection pool and on the SQL Serverthat's not good. I'll discuss connection object management in Chapter 9, where it will be clear how to get around (or, better yet, avoid) this issue.


Exploring the System.Data.SqlClient Namespace

The SqlClient namespace is responsible for all Microsoft SQL Server[3]specific operations. As I'll illustrate later, the SqlClient namespace is specifically designed for (and only for) Microsoft SQL Server Version 7.0 and later. The .NET Framework supports several similar classes used to interface to other specific data sources like Oracle (System.Data.OracleClient) and generic data sources like ODBC (System.Data.Odbc) and OLE DB (System.Data.OleDb), as shown in the Object Browser dump in Figure 8.13. Over the years, these providers have moved from place to place, but they have settled down in their current homes, so addressing them is no longer an issue. I don't expect them to move around again, but, then again, no one expected that it would take three years to get SQL Server 2005 on the streets.

[3] No, the SqlClient .NET Data Provider does not work with Sybase SQL Server. You'll need to use the OLE DB or (better yet) the ODBC .NET Data Providersunless you can find a Sybase-specific .NET Data Provider.

Figure 8.13. The Provider-specific namespaces in the .NET Framework.

Note that the System.Data.SqlClient namespace is derived from the classes in System.Data.Common. Remember the Class Diagram in Figure 8.5? Is this important to know? Yes, if you need to create a generic application that needs to address more than one backend database. Using 2.0 Framework "factory" classes, you can generate appropriate SQL Serverspecific (SqlClient) classes as well as Oracle-specific (OracleClient), OLE DB (OleDb), or ODBC-specific (Odbc) classes, as needed. I'm going to defer further discussion of this topiclook for another EBook on this feature.

Choosing the Right ProviderWhen it's Not SQL Server

When choosing the "right" data access provider, be sure to pick a "native" interface first. That is, if there is a .NET Data Provider designed to interface to your specific data source (like SQL Server or Oracle), use it. The same is true for the new SQL Server Everywhere. It uses the SqlCE namespace that must be referenced manually. If the .NET Framework does not include a suitable native provider, look for one elsewhere. There are a wealth of other providers out there sold by (or simply given away by) other vendors, like Sybase, Oracle, IBM (DB2), and others. There are also several third-party companies, like Data Direct[4] and Cor Lab[5], to name just two[6].

[4] www.datadirect.com/products/net/index.ssp

[5] www.crlab.com/products.html

[6] See www.sqlsummit.com/DataProv.htm for a list of .NET Data Providers from a host of vendors.

One big difference between providers is their dependence on COM-based components. If a provider says that it's "managed," there's a good chance that it's not COM-dependent. However, even Microsoft calls its System.Data.OleDb .NET Data Provider "managed," when it's really dependent on COM-based OLE DB providers. Perhaps the distinction is "100% managed" code, as touted in the DataDirect providers. In any case, if the provider depends on COM, it's going to be slower and more trouble-prone. That's because any references to COM components from a .NET application have to traverse a layer of plumbing code called "COM interop," which exposes COM objects so they look like .NET objects, and vice versa. While it's relatively fast, it's far from free of performance and interface-translation issues. It's also fraught with other (shall I say) "issues" that make it tough to trust. All too often. I hear about problems caused by COM interop and how it did not emulate the COM component in the same way as it executed in a COM-based application.

If you must access a data source that does not support a native managed interface, you'll have to use one of the one-size-fits-all (OSFA) .NET Data Providers like System.Data.Odbc or (as a last resort) System.Data.OleDb. The Odbc provider uses the same ODBC drivers you've been using for a decadewith all of the benefits and issues that have been discussed in earlier editions of the Hitchhiker's Guide. In my tests, the Odbc .NET Data Provider (which is not COM-dependent) is about 20% faster than OleDb. No, Odbc is not a good choice for situations where you need to access a JET[7] database, as the Odbc drivers don't expose all of the functionality you might need. This means you're stuck with OleDb when accessing JET databases. However, for any data source that doesn't have a native provider, Odbc is your best choice.

[7] JET: Remember that Access defaults to use of the JET database engine to manage its database file.

IMHO

Don't rely on my (or anyone's) testsespecially if they have the words "Sales" or "Marketing" in their title. They might not match your configuration or hardware setup. Create and run your own tests on your own hardware.


Exploring the SqlClient Namespace

Ready to drill down into the classes you'll use to access SQL Server? As I said earlier, there are two general categories of data classes you'll use to access and manage information extracted from your data source. The classes used to connect to the data source, run the query, and return the data are implemented by the .NET Data Provider, as exposed by the System.Data.SqlClient namespace. These are described next. The classes used to manage the returned resultsets in memory are exposed in the System.Data namespace, which is discussed later in this chapter.

To make the discovery and exploration process easy, this section highlights the parts of the .NET Framework namespace that you'll most likely reference in your application. Sure, many of the classes exposed by the Object Browser are used by Visual Studio and other tools behind the scenes and might not be of much use to you unless you're writing your own tools, so let's focus on the handful of namespaces you will need to referenceat least, in applications accessing SQL Server.

  • System.Data.SqlClient: This namespace exposes the SQL Server .NET Data Provider. Here, you'll leverage classes used to manage connections, execute SQL, and generate the System.Data classes used to manage the values returned from your queries and post changes back to the server.

  • System.Data: This namespace contains the objects you'll use to manage data in memory. These classes permit you to bind, change, sort, filter, find, and refine your data.

  • System.Data.SqlTypes: This namespace defines the SQL datatype structures, as well as the classes used (behind the scenes) to transmogrify a System.Type to a SqlType and back.

  • Microsoft.SqlServer.Server: When you reach Chapter 13, "Managing SQL Server CLR Executables," you'll be guided through Common Language Runtime (CLR) development, where you can explore this namespace. It contains classes used to manage connections, commands, and data passed to and from CLR procedures.

The following summaries highlight the core (and most frequently used) SqlClient classes. I discuss each of these classes in detail in later chapters, so you can look forward to far more detail and a rich set of examples.

The SqlConnection Class

When you need to access data in a SQL Server database, you need to open a "connection" to it. In the most basic sense, a connection is simply a two-directional communications link to a specific instance of SQL Server from your application. When interacting with SQL Server, a connection transmits and receives provider-unique Tabular Data Stream (TDS) packets. As long as the connection is open, SQL Server dedicates a block of memory to the connection and monitors it for traffic generated when you tell ADO.NET to run a query, fetch a resultset, or any other T-SQL or utility operation. In some cases, SQL Server and the provider also perform background handshake operations to determine if the server and client are still working. When the connection is broken (on either end), the resources used by the connection are released (at least, on the server end).

Typically, a connection to SQL Server can handle only one "conversation" at a time. However, if you enable Multiple Active Result Sets (MARS), you might (just might) be able to hold several conversations at once on a single connectionwell, sort of. While MARS permits you to have more than one operation being executed on the same connection, the resultsets from these operations are simply interleavedthe command execution is still sequential. As with a telephone party line (where several customers share the same phone line), there are specific rules and protocols to observe so that the connection can be safely and efficiently shared. I discuss MARS in Chapter 9just don't jump the gun and start enabling it before you're sure it's necessary.

Because you can't reference any SQL Server data without a connection of some kind, one of the first SqlClient classes you'll have to work with is SqlConnection. It's important enough to devote Chapter 9 to the SqlConnection class. There, I discuss how to use the SqlConnection to establish a link between a specified SQL Server instance and how to manage the connection pool used by ADO.NET to optimize the connection process. Later in Chapter 13, I show how to use a "context" connection when coding SQL CLR procedures.

This SqlConnection class is used to capture the parameters needed to establish your application's credentials, point to a specific SQL Server instance or MDF filename, set a default database, and set many other options. These option/value pairs are stored as a string in the SqlConnection object's ConnectionString property. As I'll cover in Chapter 9, the ConnectionString determines what, who, and how your application gets connected, as well as if the SqlConnection is pooled or not. In ADO.NET 2.0, the SqlConnection class also exposes the underlying schema of the target DBMS using the GetSchema method. Again, this is a feature useful for building schema-driven tools and utilities.

The SqlConnectionStringBuilder Class

The 2.0 version of ADO.NET also includes another new classthe SqlConnectionStringBuilder. Developed for use behind the scenes in Visual Studio, this class can help build a ConnectionString without having to code a string containing the correct argument/value pairs. Because the SqlConnectionStringBuilder properties are strongly typed, you'll be able to construct this crucial ConnectionString more safely. I illustrate how to use this class in Chapter 9.

The SqlCommand Class

Once the connection has been established, you must use the SqlCommand class to execute T-SQL queries against an instantiated and open SqlConnection object. Unlike ADO classic, the only way to execute T-SQL is to use the SqlCommand object. Its properties describe the query, contain the T-SQL, and include a pointer to the connection. The SqlCommand is also used to manage client-side transactions. As shown in Chapter 11, "Executing SqlCommand Objects," a SqlCommand object can return a rowset in the form of a SqlDataReader, a single object value (a scalar), a data stream, XML, and, in some cases, a rows affected value. You also use SqlCommand objects to execute stored procedures, manage input parameters, and return their rowsets, OUTPUT, and RETURN value parameters. The higher-level classes, like the SqlDataAdapter and the Visual Studio-generated TableAdapter, use the SqlCommand object behind the scenes to execute queries.

The SqlParameter and SqlParameterCollection Classes

The SqlCommand class exposes a SqlParameter class used to manage query input, input/output, OUTPUT, and RETURN value parameters. In SQL Server, the SqlCommand object's Parameters collection manages a set of named parameters that are automatically passed to SQL Server at runtime so they can be inserted into the query as they are needed. This same set of SqlParameter objects can be automatically populated with OUTPUT parameters that carry the new Identity value. I illustrate how this is done in Chapter 13.

The SqlTransaction Class

The SqlCommand class can also help manage transactions through its Transaction property, which can point to a SqlTransaction object. This way, you can tie the success or failure of one or more "atomic" queries or database DML operations with other related operations. At any time, you can instruct SQL Server to accept or roll back the changes using SqlTransaction class methods. The SqlTransaction class is also used to create ADO.NET's form of pessimistic locking cursors. These are discussed in Chapter 11.

The SqlDataReader Class

This class is used to extract raw data from the TDS stream that is returned when you execute a query that returns a rowset or BLOB stream. The SqlDataReader is unusual, in that no Microsoft data access interface since DB-Library has exposed anything like itall pre-ADO.NET interfaces have hidden this raw data stream to make data access easier (albeit less flexible) for developers. Hard-coding the SqlDataReader means you might have to handle every aspect of connecting, resultset management, and fetching for each column one at a time. While the SqlDataReader can return rows very quickly, it can be code-intensive and expensive to support if you choose to address the individual rows and columns in code. In addition, since the SqlDataReader requires manual connection management, it makes applications more prone to errors caused by orphaned SqlConnection objects, as I discuss in Chapter 9. However, since ADO.NET 2.0 exposes the new DataTable and DataSet Load methods, it's relatively easy to eliminate costly row-by-row parsing code, as I illustrate in Chapter 11.

The SqlDataReader class is implemented in much the same way as other .NET Data Providers implement their specific DataReader objects. You'll also discover that the SqlBulkCopy class introduced with ADO.NET 2.0 can accept data from any of these DataReader streamswhether they're generated by SqlClient or by Odbc, OleDb, or some provider-specific .NET Data Provider.

The SqlDataAdapter Class

When you need to create an application that fetches data from a simple table-based data stream and (possibly) update that data, consider use of the SqlDataAdapter, shown in Figure 8.14.

Figure 8.14. The SqlDataAdapter manages a set of SqlCommand and SqlConnection classes.

[View full size image]

For reasons I don't endorse or agree with, Microsoft has chosen to "hide" the SqlDataAdapter class wizards and several data-centric icons from the toolbox. While it's still fully functional, the SqlDataAdapter is now missing from its usual toolbox "Data" tab location.

Think of the SqlDataAdapter as a "super-class". It's one of the most sophisticated classes in ADO.NET and has been in place since the first version of the .NET Framework. Unlike the new Visual Studiogenerated TableAdapter, the SqlDataAdapter is a repository for a set of other classes that perform specific support tasks and supports methods to perform both simple DataTable population and update operations. The key here is that the SqlDataAdapter is designed to provide updateability to a single table's rowset, even though its DataSet is capable of containing many DataTable instances. In practice, the Visual Studio tools assume that you're populating the SqlDataAdapter, DataSet, and DataTable objects from a single table's rowset (hopefully with a subset of the table's rows). Yes, the SqlDataAdapter can fetch data from two or more tablesbuilding a separate DataTable for each rowset returned. The SelectCommand associated with the SqlDataAdapter can also execute a SELECT that returns the product of a JOIN or any number of unrelated rowsets. However, the SqlDataAdapter contains only one set of DML commands to change the target DataTable. This means when you execute the UPDATE method, you get only one set of action commands to executeeven though there might be 2 or 22 tables at the root of the rowset.

Database updateability assumes that the DBA has granted update rights to the base tablethey usually don't. This does lessen the usefulness of the SqlDataAdaptereven if you don't plan to use its Update method to post changes to the database.

The SqlDataAdapter Fill method is a powerful tool when fetching rows from several tables at once in a single round-trip. It's a great tool to populate lookup tables bound to pick lists in your UI.

You set up the SqlDataAdapter by using the Data Adapter Configuration Wizard (DACW) or by setting specific properties:

  • The Connection property points to the SqlConnection object used to perform all query and update operations.

  • The SelectCommand property points to a SqlCommand object that's responsible for fetching one or more rowsets, which usually contain a SELECT statement or a stored procedure. This SELECT is also used to auto-generate the DML queries.

  • The UpdateCommand, InsertCommand, and DeleteCommand properties point to SqlCommand objects preset to execute appropriate DML queries that update the database to reflect changes made to the DataTable object populated by the SqlDataAdapter. The SqlCommandBuilder might be able to generate these if the SELECT is not too complexunless it's a Tuesday before the full moon.

  • The UpdateBatchSize property determines how many rows are batched together for each round-trip to SQL Server. Increasing this value can dramatically improve Update performance by reducing round-trips but increase Update complexity in order of magnitude. See Chapter 12, "Managing Updates," for details.

  • The ContinueUpdateOnError property indicates whether the Update operation should stop after the first exception or continue onwhich might leave your database or referential integrity dented. This defaults to stop on first exception.

  • The FillLoadOption property determines how the Fill method deals with data values that already exist in the target DataTable objects. Since the Fill can merge data into existing DataTable objects, you can tell Fill to overwrite or preserve the current and original data values.

  • By configuring the DataTableMappingCollection, your code can map the SQL Server table names from the SELECT to the column names created for the DataTable objects.

  • The MissingMappingAction property determines how ADO.NET handles mis-matched columns.

  • When a rowset is added to a DataSet and the target DataTable or DataColumn is missing, the MissingSchemaAction property determines how Fill behaves. It can be programmed to simply add the missing column, add the column and primary key, do nothing, or throw an exception.

  • The ReturnProviderSpecificTypes property determines whether the Fill method returns provider-specific datatypes or CLS-compliant types. This property is new for ADO.NET 2.0 and is used to support the new SQL Server 2005 CLR executables.

  • The TableMappings property determines how the DataTable names map to the rowsets returned by the SELECT queries. These can be used to map the DataTable instances to database table namesassuming that makes sense.

To make the process of configuring the SqlDataAdapter easier, Visual Studio (still) includes the Data Adapter Configuration Wizard, but it (still) sets only a few of the properties listed. While it's hard to find, you can reconfigure Visual Studio 2005 to place the DACW launch icons back on the toolbox menu (as discussed in Chapter 7, "Managing Data Tools and Data Binding") and step you through the process of building a SelectCommand. From there, the DACW uses the SqlCommandBuilder to at least attempt to generate the DML queries. Visual Studio augments this code to help manage identity values and updated rowsets but does not expose any new functionality to let you alter the default (and dumb) behavior of the SqlCommandBuilder. As I'll show you in Chapter 9, the SqlCommandBuilder is very limited in its flexibility. Most developers learn to build the DML queries themselvesor simply point to appropriate stored procedures. As already discussed, the new TableAdapter configuration wizard does expose a few more CommandBuilder switches to alter how concurrency is managed.

ADO.NET and Updates

When I discuss database updates in Chapter 12, I'll show how the SqlDataAdapter is designed to handle the update chores for a single database tablea subset of whose rows are managed in an associated DataSet and DataTable. No, there is no way to get the SqlDataAdapter to handle more sophisticated operations unless you code the DML action SqlCommand objects yourself. That's exactly what I discuss in Chapter 10, "Managing SqlCommand Objects"it's not that hard.

Even if you don't plan to change the target data, the SqlDataAdapter can help make your development faster and less trouble-prone. First, because the SqlDataAdapter automatically handles the SqlConnection object for you, your code won't have to open or close the connection. The SqlDataAdapter also knows how to handle complex SELECT queries that contain two or more SELECT statementseach is used to create a separate DataTable objectas long as you don't expect to update more than one of these tables.

Introducing the SqlDataAdapter Fill and Update Methods

The most powerful features of the SqlDataAdapter are the Fill and Update methods. These take on a long list of tasks to ensure that your code need not concern itself with the clearly defined process of populating DataTable objects from rowsets. Basically, the Fill method performs the following operations:

  • Opens the specified SqlConnection (if it's not already open).

  • Executes the SelectCommand query and returns a SqlDataReader behind the scenes.

  • Using the schema returned by the SELECT, determines whether the target DataTable already exists in the DataSet. If it's already there, merges the two rowsets together; if not, creates a new DataTable object and adds it to the designated DataSet object's Tables collection.

  • Populates the DataTable with rows returned by the hidden SqlDataReader. This is actually a complex process that I'll discuss in Chapter 11.

  • Repeats this process for each resultset containing a rowset in the SelectCommand.

  • Ensures that the SqlConnection is returned to the pre-Fill stateeven if the Fill trips an exception.

The SqlDataAdapter Update method is just as sophisticated. It's designed to help eliminate considerable code when using ADO.NET to post changes made to an in-memory DataSet and DataTable objects back to the database. The Update method in ADO.NET 2.0 has expanded functionality that permits it to process several DML changes in a single round-trip to the server. This promises to reduce the time wasted making a round-trip to the server for each change made to the DataTable. The operations performed by the Update method include:

  • Scans the TableMappings to see which table is being referenced.

  • Opens the specified SqlConnection (if it's not already open).

  • Scans the DataTable object looking for rows that have been changed, added, or deleted. This is accomplished by checking the DataRow State property. If a row has been changed, the UpdateCommand is processed. Likewise, if a row has been added, the InsertCommand is processed; if a row has been deleted, the DeleteCommand is processed. If your referential integrity constraints (like primary key/foreign key relationships) require that you execute changes in a specific order, it's up to you to filter the rows being accessed by the Update method and repeat the Update method as many times as necessary to encompass changes to all parents and related children in the proper sequence (as discussed in Chapter 12).

  • This "processing" means executing the designated SqlCommand immediately (using one round-trip to the SQL Server per row), or, if batch operations have been requested, the operations are batched at the TDS level and sent in a single round-trip.

  • Once a row has been updated, the database can be queried (usually in the UpdateCommand or InsertCommand T-SQL) to determine the current value of selected columns in the database. These values are used to update the TimeStamp and Identity column(s) in the in-memory DataTable. You determine this behavior by setting specific properties of the SqlParameter instances associated with the UpdateCommand and InsertCommand. At this point, any child rows are updated to reflect the new parent identity.

  • Fire events as rows are about to be changed or after they are changed. These events can be used to perform custom operations on the data or simply derail the Update process.

  • The DataTable scan of the DataRow State property is continued until all rows have been processedas I said, this might take one round-trip to SQL Server for each change unless you have enabled batch updates.

  • Ensure that the SqlConnection (as indicated by the State property) is returned to the pre-Update stateeven if the Update trips an exception(s).

As you can see, the SqlDataAdapter is a seemingly complex beast, but it can save you a lot of time and code when working with single-table DataTable objects. When you need to fetch rows from several database tables, you have an important choice to make. As I discussed in Chapter 1, you have to decide whether or not to use SQL Server's ability to create a rowset product by joining two or more database tables, or let ADO.NET informally "join" the two rowsets using DataRelation objects. I'll discuss this issue again and cover the rest of the SqlDataAdapter functionality in Chapters 10 and 11.

The SqlBulkCopy Class

Because so many developers try to use ADO.NET to move dozens to many millions of rows from place to place, Microsoft (wisely) decided to include a new class in ADO.NET 2.0SqlBulkCopy. For the first time since DB-Library, developers can program their applications to automate the process of uploading bulk data into SQL Server without having to learn how to program BCP or SSIS. Using SQL Server's unique bulkcopy mode, the SqlBulkCopy class passes data from a SqlDataReader up to a selected SQL Server table. I encourage you to look for places to leverage its powerit can shorten a day-long upload to a few minutes. SqlBulkCopy is designed to read data from any .NET Data Provider that exposes a DataReaderthat includes all of them. This means SqlBulkCopy can import from other SQL Server systems, Oracle, DB2, flat files, or even JET.

The SqlDependency Class

Another new feature of ADO.NET leverages the .NET Framework 2.0 and SQL Server 2005's ability to notify your code when selected data changes. This means your code can execute a typical (albeit focused) SELECT command and have an event fire when any server-side changes are made to the data in that rowset. I discuss this in Chapter 11 when describing how to execute SqlCommand objects.

The SqlError, SQLException, and SqlErrorCollection Classes

When your application throws a SqlException, you'll want to explore the properties of the SqlError class to see what went wrong. The SqlErrorCollection class is used to manage the set of SqlError objects returned in a SQLException.

The SQLClientFactory Class

In situations where you need to build a generic application, you can leverage the "factory" classes to generate appropriate provider-specific classes to implement your data access interfaces.


Exploring the System.Data Namespace

Now that we're finished our initial tour of the SqlClient namespace, we're ready to move on to the .NET Framework classes used to manage in-memory datathe System.Data namespace. This namespace contains the base classes used to store and manipulate the data returned from queries or generated through whatever means you devise. Basically, the System.Data classes implement a lightweight in-memory database populated by a data source provider like SqlClient. The System.Data namespace currently supports a number of data munging operations so you can filter, sort, find, and inter-relate data. These classes also expose interfaces to permit bound controls to display the data and post changes to it, as well as fire events as the data is loaded or validated, or its state changes. When coupled with the data provider classes I've already discussed in this chapter, the System.Data classes can feed data back into the SqlDataAdapter Update method so changes and new rows can be posted back to the database.

Deciphering System.Data Naming Conventions

The Class Diagram in Figure 8.15 lays out the System.Data namespaceat least, the parts you'll use most of the time. It shows how the objects and containers are related. You'll observe that the DataSet object is used as the container for all of the other (related) objects, such as collections of Table, Relation, Constraint, Row, Column, and Property objects. Unfortunately, the .NET Framework naming convention and hierarchies shown by the Object Browser is not particularly consistent. While there are DataColumn objects in the Columns collection, and DataRow objects in the Rows collection, you might expect the Constraints collection to hold objects named "DataConstraints" to follow the symmetrythey're not. Instead, the Constraints collection contains Constraint objects, which is understandable. However, what about the PropertyCollection and ExtendedProperties? I suppose that these objects were created by Microsoft groups working in different buildings (or countries).

Figure 8.15. The System.Data namespace showing the DataSet and DataTable objects.

[View full size image]

I'm also not thrilled with the choice of a few of these object names. Yes, I like the fact that Microsoft decided on DataColumn instead of "DataField", but why call a set of rows (a rowset) a DataTable? Since a DataTable shouldn't (necessarily) map to an entire database table, but to a subset of database table rows returned by a SELECT, IMHO, it should have been called a "Rowset" object and stored in the "Rowsets" collection.

IMHO

Don't be confused by the names. Just keep in mind that you should not simply fill a DataTable with the contents of your database table just because it's called a DataTable objectnot if you expect to create an application that performs and scales well.

The next section walks you through these System.Data namespace objects and highlights basic and new ADO.NET 2.0 functionalityjust like I did for the SqlClient objects.

Understanding the System.Data.DataSet and DataTable Objects

I'm not sure it makes sense to discuss the DataSet object before you get a better understanding of what it containsbut I have to start somewhere. The DataSet is basically a container for the Relations, Tables, and ExtendedProperties object collections (exposed as properties), which, in turn, contain Relation, DataTable, Row, Column, and Property objects. As I said, the DataSet forms the foundation for ADO.NET's disconnect in-memory data architecture. This approach was initially implemented (at least, in part) in COM-based ADO classic by the "disconnected" Recordset. The disconnected ADO.NET DataTable class can be seen as simply an extension of the Recordset conceptalbeit, fairly different in content, approach, and usage methodology. Okay, they're not at all alike, except that they both hold data rows. However, instead of implementing a single stand-alone object that holds both schema and data, the DataSet object is designed to hold (virtually) any number of DataTable objects and (optionally) connect them relationally or informally on the client using Relation objects.

A DataSet can address one or more rowsets in DataTable objects, define relationships between the DataTable rowsets (regardless of their source), sort, filter, and find and save changes to the data. You can also bind to DataSet objects and the DataTable objects they contain.

This architecture is implemented such that the data residing in the DataTable objects stored in the DataSet can come from anywhereso you can "relate" data from two or more totally random data sources, if that's necessary. For example, a single DataSet can contain a DataTable object from the Customers database, another from the AccountsPayable database, and another from lower Manhattan. Your code can define relationships between these DataTable objects despite the fact they are not formally related in their respective databases. Of course, you might need their parent's permission if they are underage.

As I've said, the SqlDataAdapter can populate a DataSet, and, given the right configuration, the Update method can post changes to one of the underlying database tablesthat is, you can choose only one of the DataTable objects to update with the DataAdapter. The DataSet object can be serialized to either XML or binary (in 2.0), so it can be transported from place to place and reconstituted wherever needed. Changes made to the DataSet can also be exposed as an XML "DiffGram," in case you need to send the updates to space aliens or people in the government who understand data only if it's spelled out very simply.

Populating a DataSet with DataTable Objects

This section provides a quick look at several techniques to populate a DataSet with DataTable objects. Chapter 11 discusses these techniques (and others), including how to populate the DataSet and DataTable objects with schema and data from a variety of data sources. Once I focus on solving specific problems in the later chapters, you'll be in a better position to know which technique to use and which makes the most sense in your particular situation.

In case you can't tell, I try to use techniques that are not code-intensivethey're cheaper to develop and support and easier to learn (and teach). Actually, the amount of code the Fill and Load methods execute behind the scenes is about the same as the code you would write to perform the same functions, but it's written and debugged by the Microsoft minions, and you don't have to support it, figure out why it's not working, or hope that it is. No, this does not mean I'm a staunch fan of the drag-and-drop approaches to constructing DataSet objects, as these code-generators can lead to other issues that can waste your timeI'll get to that approach later in this chapter. That said, Visual Studiogenerated code is all well and good for many situations, but not nearly all. There will come a time when you're going to want to solve problems that the automated code generators just can't handle. I'll show you how along the way.

A DataSet data and schema (or both) can be populated in a number of ways, but they're most easily created by calling the SqlDataAdapter Fill method, as described earlier, or the new Load methods that I describe later in this section and explore in depth in Chapter 11. By "population," I mean the process of configuring the DataTable Columns collection with the schema (which defines the column names, datatypes, sizes, and other properties) and filling the Rows collection with a rowset. This can be done by hard-coding the methods and properties, but this makes your code brittlesubject to fail if the underlying schema changes or if some newbie does not understand the code. This is another argument for use of the feature-rich Fill and Load methods.

Among the techniques you can call on to populate a DataTable are:

  • Using the SqlDataAdapter Fill method. I discussed the SqlDataAdapter earlier in this chapter. This automated method uses the schema returned by the selected rowset to populate one or several DataTable objectsone for each rowset returned by the SelectCommand. Fill can also "synchronize" the DataTable object(s), in that if the DataTable being filled already contains data, ADO.NET attempts to update this client-side data with fresh data from the server based on the table's primary key (as I discuss later).

    Tip

    The Fill method requires that the DataSet be instantiatedit won't create a new DataSet for you.

  • Populating directly from a DataReader using the Load method. As I said, this technique (new for ADO.NET 2.0) exposes code used by the Fill method behind the scenes. Yes, you still have to include code to a SqlDataReader that's used to populate the DataTable. Like the Fill method, you'll need to create a target DataTable or DataSet object beforehand, but you needn't configure its schema before using LoadADO.NET can do that for you. These new methods are fast, light, and fairly easy to use.

  • Populating a strongly typed DataSet from a TableAdapter method. This is another technique that's new for ADO.NET 2.0. It leverages the Fill and Get methods generated by Visual Studio wizards when creating a Data Sourcebased strongly typed DataSet. I discussed this approach in depth in Chapter 6, "Building Data Sources, DataSets, and TableAdapters."

  • Hard-coding your own DataTable objects. This approach hard-codes the individual DataTable Column objects and populates the Rows collection manually using techniques I discuss in Chapter 11. Manual population might be your only alternative if the data does not come from a data source that can be used to build the Columns collection from the inbound rowset schema. In any case, note that the DataTable can be changed at any timeyou can add or remove existing columns as needed.

    Tip

    The DataTable can be changed at any timebefore or after rowset population. This means you can add columns or remove existing columns as needed.

  • Populating directly from a DataReader using hard-coded Get function calls. This is the most code-intensive of all techniques, and I can't really endorse it for most applications. This technique requires that you code a SqlDataReader, examine the first row of each rowset, populate the Columns collection properties, and loop through each item of each row to extract the dataa column at a time. By hard-coding the Get function calls, you make your code extremely brittleit's easily broken when the schema changes. Thankfully, this approach can be completely eliminated now that the DataTable Load method does all of this work for you.

Consider that if the Fill method is used against "empty" (unpopulated) DataTable objects, ADO.NET creates one or more new DataTable objects, populates the Columns collection, and fills in the Rows collection with data. But what happens when the target DataTable object(s) already contain data? The Fill method has to deal with this situation in several scenarios, so you'll need to provide some guidance as to how ADO.NET should deal with existing data. I'll discuss this issue and many others in Chapter 11.

A feature that you might not expect if you're accustomed to working with ADO classic Recordset objects is the ability to define a DataSet and modify it post-open. That is, even after you populate a DataSet by whatever means, you can still add, change, rename, remove, or reposition the Columns collection. This makes it easy to add special "computed" or expression columns to your DataSetwhich makes it even easier to show these special columns in your bound controls.

What's a DataRow and a DataColumn?

When it comes time to store data with ADO.NET, you or ADO.NET clone a new DataRow by calling the NewRow method on the DataTable. This method generates a new, empty row that you subsequently populate with data. This row is not part of the DataTable until you use the Rows collection Add method. A DataRow is stored as a set of object-type arrays unless you get Visual Studio to create a strongly typed DataRow. No, data is not stored behind the scenes in XML, despite what some might say.

A new DataTable can also be constructed in code. This gives you the ability to create client-side data structures of your own design. Creating a DataTable is simply a matter of instantiating a new DataTable object and populating the Columns collection. Since online help is a bit skimpy in this area, I've included an example that walks through that process. First, the DataTable needs a DataColumn that defines the primary keyan Identity column. Sure, ADO.NET knows how to manage client-side identity values just like SQL Server. The example in Figure 8.16 illustrates doing just that. Selecting a valid DataType is really pretty easyif you use this syntax. Intellisense pops up to list valid datatypesat least, in Visual Basic.

Figure 8.16. Creating a new DataColumn to act as the DataTable object's primary key.

[View full size image]

Note that I added the newly created (and configured) DataColumn to the DataTable Columns collection using the Add method. This example also illustrates how to use the AutoIncrement property. In this case, I enable AutoIncrement and set the step to 1. This way, when a row is added to the DataTable, a new Identity value is created. I also set the Unique property to True to ensure that no duplicate rows are added to the table.

The next step is to add the detail columns to the DataTable. In this case, I created an array of DataColumn objects and used the AddRange method to add them to the DataTable Columns collection. The example shown in Figure 8.17 illustrates how this is done.

Figure 8.17. Creating and configuring the detail columns for the DataTable.

[View full size image]

Each column is given a unique name as the array is populated. No, ADO.NET won't complain about duplicates until you execute AddRange. Each DataColumn is assigned a datatype, but the data is stored as an Objectas I've discussed before. The datatypes are used to ensure that the values assigned to the column's row value conform to the type constraintsinteger size, range, and so forth. Each type has its own constraints, which I'm sure you know by now. Of course, as I've discussed before, setting a datatype does not make the data pure, as far as business rules are concernedbut at least the data is type-safe.

Note that as you create DataColumn objects, you have the opportunity to set ancillary properties, such as:

  • MaxLength: The maximum length of the value appliedusually for strings. No, there is no "maxvalue" to limit the value of a numeric type (but there should be).

  • DefaultValue: The value applied to a column when the row is first created. Again, this should conform to the type constraints. I use the Now() function to return today's date to the Date-typed column.

  • Expression: For computed columns, you can pass a string that contains a valid expression. See online help for details on how these should be created. Note that the expression compiler uses a single pass to resolve the expressions, so any columns mentioned in the expression must exist higher in the Columns collection. In this case, the expression references YearsExperience and Caps but can't reference Salary. The expression syntax has a number of other quirks, including the inability to recognize Visual Basic integer divide operators ("\").

Once the array of DataColumn objects is populated, you can use the AddRange method to add it to the DataTable Columns collection. Now is when you'll throw exceptions if there is something wrong with the expressions, you have duplicate column names, or other issues arise. I usually test my newly created DataTable objects by simply binding them to a DataGridView. This way, I can test the expression handler and see how the other columns are handled.

Drilling into the DataRow Object

The DataRow object is used to hold the value for each of the columns defined in the DataColumn collection. That's a big difference from the ADOc Recordset, which stores both schema (column definitions) and data values in the same structure. In ADO.NET, most of the schema is maintained in a separate DataColumnCollection (addressed by the DataTable Columns property), and the data is stored in a separate DataRowCollection (addressed by the DataTable Rows property). Each DataRow object holds the "current" value for each column, as well as the "original" and "proposed" values. All of these values can be referenced by addressing the Item property for each columnif they exist. As you create new instances of the DataRow and set the value, or fetch a row from the database and change the value, the original value and proposed values are managed separately, so you can "undo" change made to the row. When we get to Chapter 11, on populating the DataTable, and Chapter 12, on updating, you'll see how you can return and reset these "versions" of the column value. Note that, by default, the DataRow value is stored as an object type, even though you define a datatype for the DataColumn. I discuss strongly typed DataSet objects and type safety later in this chapter.

No, don't go looking for a "Value" property on the DataRowit's not there. The DataRow value is stored in the Item or the ItemArray properties.

As you create and modify the DataRow, each instance is tagged with a RowState property that's used to indicate its "state".

  • Detached: This indicates that a new row has been created (using the DataTable.NewRow method) but is not yet a member of any specific DataTable Rows collection. If you execute the SqlDataAdapter Update method, this row is not included in any DML action command operations.

  • Added: This state indicates that the row has been appended to the DataTable Rows collection since it was built. When you execute the SqlDataAdapter Update method against this table, ADO.NET executes the InsertCommand to add this row to the database.

  • Modified: This state marks rows that have been changed since the Rows collection was populated. When you execute the SqlDataAdapter Update method against this table, ADO.NET executes the UpdateCommand to add this row to the database.

  • Deleted: This state marks rows that are to be deleted using the DeleteCommand when the Update method is executed.

  • Unchanged: When the DataTable Rows collection is first populated, the RowState is set to this state. Unless something changes, unchanged rows are ignored during the Update method.

As you can see, the RowState property is very important, as it determines whether or not the DataAdapter Update method executes one of the DML action commands or simply steps over the row as it searches for changes to post to SQL Server. Since a DataTable object's rows can be "updated" by repopulating from another rowset, you should carefully consider how this is done. Perhaps you wish to preserve client-side changes already made to the DataTable Rows collection or simply overlay them. I explore how to manage these values and other DataTable population issues in Chapter 11.

Before long, you're going to want to set the RowState property yourself instead of using the "approved" methods to do so. To make this easier (and possible), ADO.NET 2.0 exposes two new methods (SetAdded and SetModified) on the DataRow class to support changes (albeit limited) to the RowState propertyassuming the current RowState is Unchanged. If the current RowState property is not Unchanged or Added, an InvalidOperation exception is thrown. While this seems somewhat limited in usefulness, when I get to Chapter 12, I'll show other ways to set or reset the RowState property. I'll also show ways to "undelete" rows or simply remove rows from the Rows collection where their changes can't be posted to the database.

Visual Studio and Code Generation

A dramatic difference between Visual Basic 6.0 and Visual Studio .NET is the fact that Visual Studio no longer depends on "black-box" routines to implement its behind-the-scenes work. Instead, Visual Studio now generates and integrates this code into your projects to construct the interfaces it needs to perform a litany of operations. This can make your projects seem far more complex. Thankfully, Visual Studio 2005 has moved most of this generated code to partial classes that are (initially) hidden. Sure, it's more than the ADO.NET code that's tucked in behind the curtains. As you add components to your Windows Form or ASP page, Visual Studio generates the code needed to generate and configure these objects at runtime. This code is placed in separate files in your project. Part of the magic here is accomplished by using "partial" classes. New for the 2.0 versions of Visual Studio languages, this construct permits a class to be defined in any number of separate files. This way, Visual Studio can more easily generate classes that you, the developer, should be able to see (and modify), and other classes that you should not see (or modify). When it comes time to regenerate code (as when you use a wizard or change one of the diagram-based code generators), Visual Studio knows that it can alter any code in the partial classes that it has generated. It should not change any adjunct additions you have made to these classes.

Using Strongly Typed DataSets

As you should have already surmised, by default, DataTable objects generated by the Fill and Load method are "weakly typed" data structures because they store and return "objects" (untyped data)not strongly typed data elements like integers, strings, or dates. This means that the CLR language has to add considerable code to address specific data values buried deep in the object hierarchy.

Figure 8.18 illustrates two techniques to address a row value from a DataTable in a recent test. To give you a feel for the performance difference, consider that the first addressing technique (which appears in countless examples) takes about 2,500 ticks. In contrast, by simply replacing the string DataTable and Item referencing with enumerations (or integers), the same operation takes only 350 ticks. To put this in perspective, using strongly typed DataSet referencing takes about 150 ticks to perform the same operation. The point? Well, you can gain most of the performance benefits of a strongly typed DataSet by using enumerations. This technique is demonstrated again later in this chapter and throughout the book as more sophisticated examples are built. Why doesn't Microsoft use enumerations in their code generation (and examples) instead of late binding? You'll have to ask them, but I suspect it's because they think that enumerations are frail, in that they have to be independently maintained. That's very true, but if the tools helped....

Figure 8.18. Moving a row value to a TextBox using untyped data referencing.

[View full size image]

In subsequent chapters, I'll show how to use Visual Studio to build strongly typed DataSet objects, but before I go there, keep in mind that there has been an ongoing debate about whether it makes sense to create strongly typed DataSet objects. On one hand, strongly typed DataSet objects perform faster than their untyped equivalents, but as was just illustrated, you can recover most of the performance benefit by using enumerations. However, the process of creating a strongly typed DataSet is labor-intensive and still fairly brittle. That is, suppose you invest some time to build a strongly typed DataSet using the Visual Studio wizards. Behind the scenes, more than several hundred lines of complex code can be generated for a single strongly typed DataSet class. Yes, it's possible to safely tune this code, as long as you do so in your own segment of the partial class that defines the strongly typed DataSet. However, if the schema used to define the strongly typed DataSet changes, as it often does, Visual Studio 2005 overlays the previous definition with the new class. Your code in the partial class remains, but it might not match the Visual Studiogenerated strongly typed class.

Introducing the TableAdapter

As with earlier versions of Visual Studio, the 2005 version generates a tremendous amount of code to expose strongly typed DataSet objects. To this end, Visual Studio creates customized DataSet, TableAdapter, DataTable, and DataRow objects using wizards, prompted dialogs, graphical designers, and drag-and-drop techniques. This generated code inherits from the System.Data and SqlClient base classes, extending the functionality. In a TableAdapter, individual data columns are defined as strongly typed and referenced by column name. In addition, Visual Studio 2005 permits you to define relationships for the DataTable objects so you no longer have to hard-code these relationships. The schema definition for your strongly typed DataSet is stored in an XSD (XML) file that can be used to manually alter the schema or feed the DataSet Designer, as shown in Figure 8.19.

Figure 8.19. The DataSet Designer showing four related tables from the Biblio database.

To create this relationship diagram (and the code behind it), I simply created a project Data Source that pointed to the Biblio database and four tables that I knew were related. Visual Studio did the rest. Once the code was generated, I right-clicked on the generated Data Source and chose "View in Designer". Note that the relationships between the tables have been discovered and incorporated into the graphic display. In addition, each DataTable includes a custom "Fill, GetData()" item that can be customized to alter the initial SELECTas when you wish to define a custom query to incorporate a parameter-based query or a stored procedure. You can also add as many of these items as you needthey are exposed as methods on the strongly typed TableAdapter generated behind the scenes. If you recall, I focused Chapter 6 on generating strongly typed DataSet objects, including the TableAdapter.

The advantages of this approach include:

  • The code runs much faster, as the need for converting strongly typed inbound data values to the stored type is not necessary.

  • The generated DataRow objects are "type-safe"they won't permit the code to set incorrectly typed values.

  • The relationships are generated automatically (unlike the DataAdapter wizard in previous versions).

  • The fetch mechanisms are far more flexible, in that you can easily define or tune your own custom Fill or GetTable methods, including incorporation of parameter-driven queries and stored procedures.

  • This metaphor can be used to generate a complete data access UI, including parameter capture; add, save, and delete buttons; as well as a bound DataGridView showing the data (as shown in Figure 8.20).

Figure 8.20. Binding the TableAdapter to a DataGridView using drag-and-drop.

However, to be clear, strongly typed DataSet objects are brittle. If the underlying schema changes, you usually have to go back to Visual Studio to alter the generated code. Yes, this means going through the wizards again and repeating the drag-and-drop operation used to initially generate the codeunless you dig into the XSD file, make your changes there, and get Visual Studio to regenerate the code. Then again, there are those who would counsel that if the schema changes, you'll still have to revisit the code. I agree, but the tools don't really help identify what parts have changed until you regenerate the TableAdapterthen you have to track down all of the compile errors. Perhaps that's an advantage.

The DataSet and DataTable Behind the Scenes

The ADO.NET base-class DataSet is designed around ADO.NET's "disconnected" architecture. It's designed to store and help manage data in separate DataTable objects without requiring a persistent connection to the data source(s) used to populate it. While it might (just might) be necessary to connect to a database or other data source to populate the DataSet Tables collection, once the rowsets have been moved to the DataTable (a process called rowset population), the connection can be (and probably should be) closed. At this point, the DataSet can be sorted or filtered (via a DataView), subsets of rows can be selected via property settings, new DataTable objects can be generated, additional DataTable objects can be incorporated, and the entire DataTable can be serialized to another tier. Serialization is an important feature of the DataSet and (now in 2.0) the DataTable objects. It permits data to be extruded (serialized) in XML or binary (in 2.0) so it can be transported to another application, layer, or tier, or persisted to a file.

As I've said before, DataTable data is stored in binary arrays, one for each column in the DataTable. Sure, the DataSet can be populated by importing XML into its base DataTable objects using the ReadXml method, and it can extrude XML using the WriteXml method. You can populate a DataSet schema from XML with the InferXmlSchema and ReadXmlSchema. However, it's unwise to equate XML with performance. While XML is extremely flexible, it's also very (very) verbosekinda like listening to some senator drone on and on about his pet project when the same information could be communicated in a few words. XML data structures are, by their very nature, heavy consumers of RAM, CPU, and I/O resources, as they must be "compiled" into intelligent data on each access. No, they should not be used where you don't need "universal" or extremely flexible data access paradigms.

A DataSet is capable of storing a large number of rowseven millions of rowssubject to the RAM constraints of your system and the sanity of the design. While storing even thousands of rows does not make sense for most applications, I understand that there are situations where it's necessary (albeit imprudent) to store that much information in memory. In this regard, ADO.NET 2.0 has dramatically improved performance when seeking, serializing, and editing data within giant DataSet objects. This can make it easier to implement "challenged" designs.

Unlike the ADO classic Recordset, the DataSet is designed to manage several rowsets at once and manage them individually in DataTable objects. This means you can build a DataSet from one or several data sources, add relationships between the DataTable objects, and perform a number of powerful parent/child operations. For example, you can populate a DataTable with rows from the Customers table and another containing customer orders. By adding a DataRelation that specifies the primary and foreign keys, your code can inform ADO.NET that these two tables are related and should be treated as such. This means you'll be able to fetch related orders for selected customers or find the customer for a given order. Yes, the relationships can be defined several layers deeprelating customers with orders with items and so on.

While earlier versions of ADO.NET forced you to define these relationships in code, the Visual Studio 2005 IDE now supports functionality to build the XSD (and XML structure) for you that includes these relationships. It can also generate the code to expose these hierarchical data structures as strongly typed DataSet objects. I discuss these IDE features and how they fit into an overall development plan in Chapter 6.

A single DataSet object's DataTable changes can also be posted to the database when managed by a DataAdapter or TableAdapter objects. By defining the T-SQL commands to perform the UPDATE, INSERT, and DELETE operations (what's called DML[8] or action commands), you can program ADO.NET to pass these parameterized commands to SQL Server to post client-side changes to the database. Yes, Visual Studio can help create these action commands if they aren't too complex. I discuss update techniques, the CommandBuilder, and associated issues in Chapter 12.

[8] DML: Data Manipulation Language. The T-SQL commands used to manipulate the database to change the data or the database structure.

Exposing Data with the DataView Class

One of the most powerful (and useful) properties of a DataTable is DefaultView. This property returns a DataView object that's rich with functions, methods, and properties of its own. Since the DataView extends the properties of a DataTable, you'll find the DataView useful to sort, filter, and find rows in a variety of ways. Note that the DataView is simply thata "view" on the target DataTable. Changes made to the DataView rows are applied to the DataTable. Typically, once a DataTable is created, you can bind to the DataView. This way, when you change the properties of the DataView, the rows viewed in the bound control change as well. For example, changing the RowFilter property shows only rows based on the specified filter (such as "City = 'Redmond'"); only rows where the City column contains "Redmond" are displayed in the bound control.

The most interesting of the functions, properties, and events exposed on the DataView include:

  • The Find function locates a row in the DataTable matching specified sort keys.

  • The FindRows function returns a set of rows from the DataTable that matches the specified sort keys.

  • The ToTable function (new for 2.0) returns a DataTable based on the DataView selected rowset. This means you can use the RowFilter or RowStateFilter to select a subset of the rows in a DataView and return a new DataTable based on those rowsvery cool.

  • The AllowDelete, AllowEdit, and AllowNew properties determine whether changes are enabled or disabled on the DataView.

  • The ApplyDefaultSort and Sort properties determine how (or if) the DataView rows are sorted. Yes, you can choose ascending, descending, or multiple-column sorts.

  • The RowFilter property works like a SQL WHERE clause. It determines which rows are visible in the DataView.

  • The RowStateFilter property filters the rows in a DataView and can permit your code to return only added, current (unchanged, new, and changed rows), deleted, modified, current, original, or unchanged rows. This permits you to expedite the process of updating specific subsets of the rows in the DataView.

  • ListChanged Event fires when rows included in the DataView change.

To JOIN or Not to JOINThat's The Question

One final point I would like to make before I move on. Ever since ADO.NET appeared, developers have asked one important question: "Should I fetch individual tables into a DataSet object and add the relations (or use Visual Studio to do it), or should I execute queries on the server that join the tables into a combined JOIN product and try to manage it in a single DataTable?" Unfortunately, the answer is not simple, so I need to spend some time discussing the issues so you can make an intelligent choice.

When you choose to return multiple rowsets into separate DataTable objects, you can subsequently set up DataRelation objects to handle the relationships between the DataTable objects. As I've shown you, if you set up a Data Source and TableAdapter, Visual Studio builds these relationships for you. When it comes time to update the data, you'll have to set up a specific set of SqlCommand objects for each DataTable you intend to changeno, I don't expect that all of the rowsets you fetch will necessarily need to be updated on the server. Again, with Visual Studio 2005, the TableAdapter is built and programmed to include basic DML commands to post changes to the database. These updates must follow a specific sequence to maintain the parent/child relationship constraints. That is, you'll have to add new and change existing parent rows first, followed by adds, changes, and deletes to child rows, followed by deletes to parent rows. I detail this approach in Chapter 12. However, it might not work at all with a de-normalized table or a schema with a unique parent-child hierarchy.

The problem with this approach is that folks tend to return too many rows to populate these DataTable objectsoften entire database tables. They then wonder why the performance is so poor. If your code focuses the rowsets from the task at hand and not the entire population of New Zealand, you should be able to get away with this approach. This approach can also be used with multiple stored procedure callseach returning one or more rowsets that are mapped to individual DataTable objects. However, in this case, you won't be able to use the SqlCommandBuilder or Visual Studio's tools to generate the DML queries to change the data. In most cases, you'll have already created stored procedures to perform these operations anyway, so this should not be an issue.

When you choose to perform a server-side JOIN and return a combined product as a rowset, you're taking advantage of the indexes, query engine, and processing power of SQL Server (that you paid for). This approach is most common with those DBMS rigs that are built on a stored procedure foundation. In reality, when you build a DBMS around stored procedures, it's unusual to grant access to base tables, as all database I/O is done through protected stored procedures. Yes, you can set up DataRelation objects on these DataTables, and you get all of the benefits of this approach as if the rowsets were drawn directly from database tables. However, since the rowsets are composite products from several database tables, this approach might not make much sense. Updating these DataTable objects will also be more difficult, as you'll have to determine where the data came from in the first place. However, you can simply leave that up to your stored procedures. As your DBMS gets more sophisticated, being able to perform targeted table updates (as supported by the drag-and-drop code generation) is increasingly rare. The update procedures are often complex sets of logic that change not one, but several related tables and perform far more operations than ADO.NET can handle on its owneven with your help. Let's defer further discussion of update strategies until Chapter 12.


Summary

This chapter reintroduced you to ADO.NET 2.0. Yes, I assumed that you know a bit about ADO.NET and perhaps a bit about ADO classic, but I think I was able to fill in the essential gaps pretty well. I showed you how Visual Studio and the languages can be used to expose and instantiate these objects, and how they can be referenced in code. Yes, virtually all of these objects and their properties, methods, functions, and events are discussed and illustrated with examples in later chapters.


Chapter 9. Getting Connected

    Introduction

    Connectivity StrategiesThat Include Security

    Configuring the Server and Firewall

    Connection Strategies

    Establishing a Connection

    Writing Code to Create Connections

    Understanding and Managing the Connection Pool

    Building a ConnectionString for Other Providers

    Getting Visual Studio to Build Your ConnectionString

    Opening and Closing Connections (Semi-) Automatically

    Understanding the Connection Properties

    Using the Connection Methods

    Handling Connection Events

    Managing Connection Exceptions

    Summary


Introduction

One of the first chores your code will have to accomplish is getting connected to a selected SQL Server instance. Given the number of newsgroup messages I see every day regarding connection issues, it makes sense to provide quite a bit of detail here about the challenges you'll face. Here, I'll talk about a number of SQL Server configurations and installation options and how each requires different connection strategies. Connecting to more recent versions of SQL Server has been made especially difficult, given the new default configurations that essentially make SQL Server "invisible"instances no longer broadcast their existence or even enable external ports or protocols. I talked about how to configure SQL Server to make it visible on the network in Chapter 2, "How Does SQL Server Work?", and I'll add more detail on these issues here in this chapter.


Connectivity StrategiesThat Include Security

According to a number of researchers at the University of California at Berkley, on January 25, 2003, at 05:30 UTC, the now notorious "Slammer" worm exploited a buffer overflow on systems running SQL Server or MSDE 2000. Even though Microsoft had identified and provided a patch for the problem in July 2002, over 75,000 SQL Server systems were infected[1], and, in many cases, their data was destroyed. Ever since the Slammer worm hit, Microsoft has spent considerable time and energy working on ways to prevent attacks of all kinds. While SQL Server 2005 is shielded from the Slammer worm and a host of other threats, one additional issue Microsoft's developers addressed in the current versions was SQL Server's 2000's inability to be easily patched when new vulnerabilities arise.

[1] www.caida.org/publications/papers/2003/sapphire/sapphire.html. "The Spread of the Sapphire/Slammer Worm."

Tip

All versions of SQL Server 2005 now have the ability to be patched from the webjust like Microsoft Office applications and Windows itselfeven the versions you install on the user's desktop.

Slammer took advantage of SQL Server's default setup configuration that exposes port 1433 to the network, as well as exploiting a buffer overflow vulnerability that had been left behind in unpatched systems. Add to that the tendency of Systems Administrators (at least, who that don't care enough about security) to install older versions SQL Server with a blank SA password, and you have a formula for disaster, regardless of any buffer overflow patches that Microsoft inadvertently left behind. Since SQL Server 2000 SP3, SQL Server's default configuration hides itself behind a wall of protection schemes, and it encourages use of strong SA passwords (but they still aren't requiredeven in SQL Server 2005). When you configure SQL Server's protection schemes to permit external connections, you must disable some of these protection mechanismshopefully, not all of them. I recommend that you expose just enough "surface area" to permit your users to access the systemand no more.

IMHO

Exposing SQL Server to your network as opposed to simply exposing it to the local machine significantly increases the surface area of attack for those with malicious intent.

Choosing a Connection Strategy

Before you get started, you need to make sure that your SQL Server is prepared to accept connections. Consider that SQL Server can be installed in a number of ways and on a number of Windows platforms. Each type of installation uses a different mechanism to protect itself from the outside world. Your code won't be able to connect to these servers (at least, not from remote systems) unless you reconfigure SQL Server to expose its ports, enable its protocols, and (most importantly) poke a hole in your firewall(s) to expose the SQL Server ports and protocols to the net. No, that does not mean you need to expose your server to the entire intranet or to the web.

As you start working with Visual Studio, you might not encounter an ADO.NET connection string at all or have to worry about what it contains. That's because Visual Studio can construct one for you, as I'll discuss later in this chapter. However, when something goes wrong (and it usually does), knowing what the ConnectionString contains and the options it's using to connect can be a valuable tool when debugging your application's ability to get connected. The new Data Sources dialog in Visual Studio does just thatit builds a ConnectionString and an ADO.NET Connection object to match the provider you specify, along with a host of other support objects based on the database objects you associate with the DataSource. But I'm getting ahead of myself. I'll discuss the Data Source dialog and all of its options later in this chapter and subsequent chapters.

There are three types of connection strategiesevery version of SQL Server can be configured to support these strategies. No, not all versions of Visual Studio support all of these strategiesI'll show you which are and which aren't supported. Note that each SQL Server instance can be configured independently to expose or hide itself from the network. By default, SQL Server is installed with all of the protocols and ports needed to connect to external clients disabled. However, even when ports and protocols are enabled, that's not enough to let you connect to SQL Server. You'll also need to have proper credentials and be able to penetrate your system's firewall if you want to access an intranet or web server instance. Let's look more closely at the various connection strategies.

By default, SQL Server is hidden from the intranet (your network) and the Internet (the entire world). You'll have to reconfigure it after installation to make it visibleif you dare.

  • Local access: When your SQL Server is installed on the same system as your Windows or web-based application accessing SQL Server, you don't need to do much reconfiguring to connect once SQL Server is started. Typically, a SQLExpress instance runs on the same system as the application (but it, too, can be exposed to the network). When connecting to any locally running instance of SQL Server, ADO.NET can use the "shared memory" provider that bypasses the network protocols, and your server does not have to expose the network ports to permit connections. When accessing a local SQL Server instance, there is no need to disable or alter the Windows XP or your own firewallit can keep blocking access to the system's (and SQL Server's) ports.

  • Intranet access: When you want to expose SQL Server to clients on an intranet (a local area network, or LAN), you have several more hurdles to jump before a SQL Server instance is going to permit connections. In this case, it's best to use the SQL Server Surface Area Configuration tool to make sure your SQL Server is properly configured. This configuration is used for client/server or ASP installations where the server does not reside on the same server as IIS. To access a networked SQL Server, you'll need to enable one of the external protocols (shared memory won't help for external connections), as well as ensure that the correct TCP/UDP ports are visible through the firewall.

  • Internet (web) access: If you must, you can expose your TCP ports to the Internet. This will permit anyone (anywhere in the world) to snoop the ports to "see" your server and attempt to penetrate your only remaining shieldthe password on the administrator's account. While I don't recommend this approach, I'll show you how to open a connection to a SQL Server via its public (or private) IP address and port. Just don't come crying to me when your database is compromised by some guy in Whoknowswhereistan.


Configuring the Server and Firewall

Using the Surface Area Configuration Tool

The SQL Server 2005 Surface Area Configuration (SAC) tool (shown in Figure 9.1) is a new (and welcome) tool used to simplify the task of managing and securing your SQL Server instances. It can be run on most any system and be used to connect to and configure SQL Server 2005 instances on the local or remote systems.

Figure 9.1. The Surface Area Configuration tool.

[View full size image]

Using the SAC tool to manage how much (if any) of your SQL Server Instances is exposed to the LAN or WAN is easy.

1.
Choose the server to configure by clicking on "localhost"[2]. This exposes a dialog that permits you to address another server. Nope, this won't work against older versions of SQL Server or servers behind a firewall.

[2] Don't be confused. When you create a ConnectionString and want to refer to a locally running SQL Server instance, you can use "(local)" or simply "."not "localhost".

2.
Click on "Surface Area Configuration for Services and Connections". This exposes a dialog reflecting the state of all SQL Server instances on the targeted server. As shown in Figure 9.2, this server has three instances installed: SQLEXPRESS, SS2K5, and SS2K5_2. SAC is one of several tools that can be used to start, stop, or pause a SQL Server instance or any of the associated Windows services, like Integration Services or Reporting Services (when installed). Here is where you can specify whether the server is to be automatically started when the system bootsset the "Startup type" to "Automatic."

Figure 9.2. Surface Area Configuration for Services and Connections.

[View full size image]

3.
Click on "Remote Connections" in the left dialog (shown in Figure 9.3). Note that the default setting permits only local connections. To make SQL Server visible to the local intranet, you can choose either TCP/IP or Named Pipes (or both) protocols. Which is better? Generally, I prefer TCP/IP, as it's more universally supported and understood. Let's just say that Peter and I have given up trying to make Named Pipes workit seems with firewalls and other considerations, it is just simpler to concentrate on TCP/IP protocols.

Figure 9.3. Select a specific SQL Server instance to inspect or modify its properties.

[View full size image]

This step must be repeated for each SQL Server instance you want to expose to the intranet (LAN). Note this dialog does not permit you to change the TCP/IP ports that SQL Server uses to listen to the Net. These can be changed (if really necessary) by using the SQL Server Configuration Manager. Be sure to restart the SQL Server instance(s) after you make changes to the enabled protocols. This can be done in the SAC tool or the SQL Server Configuration Manager, or using the Services.msc console application.

Static vs. Dynamic Ports

As I said, the default instance of SQL Server listens for incoming connections on port 1433. The port can be changed for security reasons or because of a client application requirement using SQL Server Configuration Manager. By default, all SQL Server named instances (including SQL Server 2005 Express Edition) are configured to listen on dynamic ports. To configure a static port, leave the TCP Dynamic Ports box blank (it defaults to "0") and provide an available port number in the TCP Port box in the TCP/IP properties settings dialog exposed by the SQL Server Configuration Manager. See Figure 9.4.

Figure 9.4. Setting "dynamic" TCP/IP port assignments.

If you don't select dynamic port addressing and you choose TCP/IP protocol, SQL Server (by default) uses TCP port 1433 and UDP port 1434. This means you'll need to poke a couple of holes in your firewall to permit data to be sent and received on these ports. In this case (as shown in Figure 9.5), I created two exceptions: one to permit TCP port 1433 and another to permit UDP port 1434 to pass through the firewall. I also chose the "Change scope..." option to limit visibility of these ports to my local subnet. This prevents the ports from being visible on the web.

Figure 9.5. Using the Windows XP Firewall Exceptions tab to enable ports.

In a similar way, if you choose the Named Pipes protocol, you'll need to enable port 445 on your firewall in addition to ports 1433, 1434, and port 135[3], as discussed earlier when using TCP/IP protocol.

[3] See http://support.microsoft.com/kb/841251/

To test this functionality, you can start Visual Studio and use the Server Explorer[4] to create a new Data Connection. If this works, you know your settings are working. You can also use the new SQLCMD utility (that ships with SQL Server 2005) to test the connection (as shown in Figure 9.6). Note that I passed the server\instance name (betav1\ss2k5) and the Login ID as "SA"the arguments are all case-sensitive. SQLCMD prompts for the password; if it can connect, it (quickly) returns "1>" (the prompt for a line of SQL). To exit SQLCMD, enter "exit" and press Return. If you have not configured SQL Server to support "mixed-mode" security (which permits you to authenticate a connection using a valid SQL Server Login Id and password), you won't be able to log on as SAeven if you have stolen the password from the DBA's desk.

[4] Note that the "Server Explorer" is called the "Database Explorer" in the Standard and Express versions of Visual Studio .NET 2005.

Figure 9.6. Using SQLCMD to verify that you can connect to a specified server.

[View full size image]

Connecting to SQL Server 2005 from older clients is possibleeven using the older MDAC stack. However, the exception/error messages you get when the server is inaccessible behind a firewall or is not configured for remote connections can be very different.

If SQLCMD can't connect for some reason, you'll get a set of exception messages, as shown in Figure 9.7. Note that SQLCMD is using the new SQL Native Client (SNC) providerthat's why it's handy when you're trying to isolate provider issues. For example, SNC might work when the .NET SqlClient managed provider doesn't for some reason.

Figure 9.7. When things go wrong and you can't connect.

[View full size image]

Internet Access

Okay, for some reason you want to expose your SQL Server to the Internet. While this is not recommended, a subtle change to your firewall settings can expose your SQL Server to everyone on the planet with a computer. While your firewall might be different, the Windows XP firewall (installed with SP2) permits you to change scope for your port assignments. When you click Add Port (and fill in the port settings) and "Change scope...", you can choose to expose the selected port to any computer, including those on the Internet (as shown in Figure 9.8). Actually, this is the default setting, so be sure to choose "My network (subnet) only" or enter the IP addresses of selected systems in a custom list.

Figure 9.8. Exposing a port to the Internet.

[View full size image]


Connection Strategies

Either your application architecture dictates which connection strategy you must use or best practice does so. You'll find a lot of advice out there about how and when to connect (and disconnect). Here, I'll show you the trade-offs and advantages of each approach. Consider that getting connected is expensiveeven when connection pooling is enabled (and it is by default). If you're using SSPI authentication, each time you connect, the "user" credentials must be revalidated. When connecting to SQL Server, the server must reset the connection when it's first openedand again each and every time it's reopened (unless you disable this feature). Since connecting to a SQL Server instance is not free, you should take steps to minimize the number of connections you make if you're concerned about performance. That said, remember that SQL Server connections are a limited resourcebut not that limited. Of course, your SQL Server license might dictate how many user connections you can expose, so that number would be your upper limitand dictate how you manage connections. If you don't have a licensing restriction, you should be able to connect a thousand or more clients to the server if your code manages what's getting executed on these dedicated connections. Even though a user is connected, their client application might not be executing query or update code, so the impact on SQL Server is simply the RAM resources and a few thread cycles used to poll the status.

SQL Server has a finite capacity to perform work. You know that, but you can't easily tell how that capacity relates to the number of customers, clients, hits-per-second, or other load metrics without considerable testing. Just because SQL Server can support thousands of connections does not mean it can handle a thousand simultaneous ASP pages or active client applications demanding attention at once. At one point in time or another, your server might reach capacityhow your application(s) behave if that happens is a critical factor in its success. Having connections time out when you try to open them is a symptom of either sloppy connection object management (that I discuss later in this chapter) or poor query design, or the server cannot handle additional volume.

Let's look at a few connection strategies and how you should approach application connection.

Connecting Just in Time

When working with ASP applications or XML Web Services, (generally) your connection cannot be maintained between invocations of your code[5]. That is, in a typical ASP (or ASP.NET) application, your page is loaded, your code opens a connection, you perform any queries or updates, and you close the connection before your page closes. However, this execution pattern does not mean you need to open your connection several times in the course of processing a pagein fact, you shouldn't. If your code can ensure that any open connections are closed before the page is unloaded, it's perfectly okay to leave the connection open if you anticipate having to reuse the connection later in the application.

[5] Yes, it's possible to create ASP or middle-tier applications that maintain persistent connections, but these are very difficult to manageI don't recommend this approach for the feint of heart.

"Just-in-time" (JIT) connection strategies rely heavily on the connection pool to optimize performanceeven though the pooler is robbing your application of some performance. When using JIT connections, the pooler keeps the SQL Server connection open in anticipation of your application returning to use it in the immediate future. The problem with using the connection pool is that you need to (constantly) watch it for overflow issues, for reasons I just mentioned. Remember, the connection pool should not grow linearly (or exponentially) past its optimum performance plateau. The number of pooled connections should remain relatively flat once the system reaches stasis[6]when the number of connections being demanded matches those ready to be used in the pool.

[6] Stasis: a condition of balance among various forces.

Persistent Connections

When building client/server applications, you don't really need to use the connection poolespecially if your application is designed to open and hold open a connection for the life of the application. I've been designing applications that use this strategy for decades. Another benefit of this approach is that you can set and maintain server state on a connection-by-connection basis. This means you'll be able to implement server-side cursors, manage #temp tables, make SQL Server SET option changes, and much more. If you're using the connection pool but want to leverage server state, you might find it very frustrating as the default behavior resets the connection (and resets any server state) when it's reopened. Yes, you can disable this behavior, as I describe later in this chapter (see the ConnectionString options).

The trick to using an application-shared connection is to serialize its use. That is, (unless you use MARS as discussed later) you need to make sure that only one thread in your application is using the connection at any one time. For example, if your application has several forms, they can all use the same connectionbut only one at a time if you leave the connection open.

One approach to managing connections is to create a routine that opens the connection if it's not already open, prevents other threads from using it, and closes it if it's idle for a period of time. However, if you need to go to this trouble, you should reconsider using the connection poolerit does all of this for you. The other problem with a home-grown connection-management scheme is that the ADO.NET methods that manage connections on their own won't use itthey simply call the Open method on their own. No, I'm not a fan of manually managing connections. It's a lot safer, cheaper, and easier to just use the connection pool. That said, if you plan to use server-side cursors as described in Appendix IV, "Creating and Managing Server-Side Cursors," you'll want to use a persistent connectionone that's left open while the cursor is being accessed.

Understanding MARS and Its Implications

Multiple Active Resultsets (MARS) is a new feature in ADO.NET 2.0 that you might never need. Fortunately, it's disabled by default[7]. MARS permits developers to execute multiple queries on the same physical connection. ADO.NET accomplishes this behind the scenes by creating up to nine "logical" connections per physical connection. Each of these "sessions" is managed (more or less) independently. If you attempt to execute more than nine operations on the same connection, your performance will degrade (dramatically).

[7] The SqlConnectionStringBuilder enables MARS by default. Be sure to reset MultipleActiveResultSets to false to disable this feature.

One benefit of MARS is that it might prevent you from seeing the troublesome "There is already an open SqlDataReader on the connection..." exception. The problem with hiding this exception is that it's telling you that you have left an uncompleted operation behind. Enabling MARS to solve this problem is like taking out the bulb in the engine oil low-pressure indicator. MARS does not help performanceit adds overhead to the connection pooling mechanism. As a matter of fact, if you make a mistake and overload MARS with more than ten operations, your application might appear to lock up as MARS careens into the ditch trying to accommodate the load.

Why did Microsoft do this? Well, IMHO, MARS was invented so Microsoft could add a "check" to the box in the Oracle vs. SQL Server marketing material that said it supported multiple operations per connection. Another (apparently) good reason to use MARS is that it permits you to open a connection, run a query to fetch rows, and use the same connection to post updates to the row. Since both operations can now share a common transaction, there is some benefit to this approach. However, the approach itself is discredited. Most applications that move rows to the client to perform updates in this manner can/should be rewritten to perform these operations directly on the server with a smart WHERE clause or a stored procedure.

IMHO

MARS: Just say no.

Choosing the Right Scope for the Connection Object

In your application's design phase, you need to decide how many forms, pages, or classes need access to a common connectionif any do. Remember, connections are a "semi-precious" commodity, like gas 200 miles from the nearest station, but unless you plan to scale your application to support thousands of users, it's probably okay to use multiple "local" connections rather than one central connection.

A local connection is one that's opened in the routine that needs a connection. A "global" connection is declared as a SqlConnection object in a scope that can be accessed by all of the routines (forms, modules, classes) in your application. No, you can't usually get away with globally scoped SqlConnection objects in an ASP or middle-tier applicationthey make the most sense in a Windows Forms application.

You're asking for trouble if you instantiate a SqlConnection in a function and pass out a SqlDataReader feeding off the open SqlConnection. As I'll show later in Chapter 10, "Managing SqlCommand Objects," you can program the SqlDataReader to close its associated connection when (or if) the SqlDataReader is properly closed. Based on my experience, the problem is that developers don't always close the SqlDataReader when they're done with itor the SqlConnection associated with it. That might be because the consumer of your function (that returns a SqlDataReader) is another person or team that does not realize that it's their responsibility to close the SqlDataReader once they have finished reading the data. The problem is, the code that accepts a SqlDataReader as an argument does not have direct access to the SqlConnection used to feed the SqlDataReader, so they can't close the underlying SqlConnection even if they remember to do so.

No, it has never been possible to pass a live SqlConnection object between processes. This has more to do with the fundamental architecture of the low-level connectivity interfaces than anything else. While it's entirely possible to pass a ConnectionString from tier to tier, you won't get away with passing a populated SqlConnection instance.


Establishing a Connection

The process of connecting your ADO objects to SQL Server can seem like making a long-distance call to a remote island near Fijiunless you happen to live in Fiji, when it's more like getting a call placed to Kabul, Afghanistan by way of Lake of the Ozarks. The connection process is impeded all along the way by security checks and unreliable "carriers"the network hardware, protocols, handshakes, and interfaces. Even if you gain access to the server hosting SQL Server, you face another set of security checks and pat-downs once you try to check into the database. Sometimes, it seems like SQL Server is trying to look for toothpaste in your luggage. Fortunately, when connecting to SQL Server, you won't have to put up with long lines, cramped seats, terrible food, and surly flight-attendants just off of their latest strikeif you're lucky.

A successful connection assumes that the SQL Server process is actually running. In a typical shared SQL Server installation, this might (just might) be a given, but in a SQL Express installation, it's likely that SQL Server will not be startedthat will be up to you and your code. This call also assumes the network is up and stays up for the duration of the connection. Anyone with a modicum of experience knows that lots of stuff can happen to the NIC card and the cables, hubs, routers, switches, ISA servers, firewalls, and modems that carry you to the destination and the equivalent hardware infrastructure on the server end of the wire. Microsoft provides a few new helper classes in the 2.0 Framework to discover whether the server and network are still viable. However, these classes do not automatically code the exception handlers in your application to deal with the stuff that happens. I think Microsoft could have done more...but there always seems to be a next version on the horizon. For example, the SqlConnection object has a new State.Broken enumeration on its State property. Sadly, this is not used (for anything), and when the connection to the server is broken, the StateChange event does not fire (except if your server is moved to Ohio). Wouldn't it be nice if this event fired when the server went down or a stored procedure closed the connection from the server end? The problem is, the data layers don't know that the server has failed without (expensive) polling. This means there is no way of knowing (with today's technology) that a "circuit" is dead without trying to use it. I don't expect this to be fixed until some fundamental changes are made to the lowest-level network infrastructure. Don't hold your breath.

Finding Servers and Instances in your Domain

If you're writing a tool (like Visual Studio's Server Explorer), you'll need help finding SQL Server instances or data sources exposed by other providers. You might also want to determine whether the targeted system has SQL Server installed or running. The 2.0 Framework includes several new classes that can make this easy (or, at least, easier). Once you find the SQL Server instance, you'll also want to know whether it's running, and if it's not, you'll want to start it. Let's walk through the new classes to help in each of these tasks. Too bad there aren't equivalent methods to get through airport security.

The example shown in Figure 9.9 [8] retrieves a list of SQL Server instances on the domain. It uses the DbProviderFactories.GetFactory method to find all servers that are visible to System.Data.SqlClient. This returns both SQL Server 2005 (Version 9), SQL Server 2000 (Version 8), and (theoretically) SQL Server 7.0 servers and installed instances. Behind the scenes, these classes make two "calls" out to the networka SQL Serverspecific mechanism (SSRP) and NetServEnum. The former returns the version number and clustering info; the latter (if it works at all) returns just the server (machine) name. Don't be impatientthese calls are notoriously slow.

[8] This application can be found on the DVD under "ListSQLServerInstances". The example also includes code to pause, continue, and stop selected SQL Server instances.

Figure 9.9. Capturing SQL Server instance names and status.

I added code to test the status of these instances so you can see at a glance which are running, paused, or stopped. Note that the code uses "MSSQL" as the service name root (if there is an instance name, it's concatenated with "$<instance name>".

Once you've found the right SQL Server service and instance (what I call the ServiceName in the example in Figure 9.9), you can ask the System.ServiceProcess.ServiceController WFC class to start, pause, continue, or stop the service. Ah, you need to be careful here. Arbitrarily stopping a service is not a particularly good way to stop SQL Server. While it may be safe to pause, continue, or start the server using the technique I'm about to show you, unless you know SQL Server is in a stable, idle state, it's never a good idea just to stop its service. While the server knows how to protect itself and shut down gracefully, it does not notify other users that the service is going down and that they need to commit their work. Using the T-SQL command "SHUTDOWN," as shown in Figure 9.10, does about the same thing. Sure, you can execute this from ADO.NET using a SqlCommand object as well. No, it won't be possible to use SQLCMD[9] to restart the server, as you won't be able to connect to it.

[9] Remember, SQLCMD is simply the SQL Server 2005 version of OSQL that replaced ISQL.

Figure 9.10. Using SQLCMD to shut down SQL Server.

[View full size image]

The following example (shown in Figure 9.11) shows how to start a specific SQL Server ServiceName. As discussed earlier, the service name is made up of the Microsoft SQL Server service name root "MSSQL" concatenated with a "$" sign and the instance namefor example, "MSSQL$SS2K5".

Figure 9.11. Starting a specific SQL Server instance on a named server.

[View full size image]

To see which services are running (and provide yet another way to start, stop, pause, or configure any Windows service, you can use the Services.msc snap-in application, as shown in Figure 9.12. In this case, I drilled into the SQL Server (SS2K5) instance properties page. Note the "service name" is shown at the top of the dialog. This is the name you pass to the WFC class used to alter or view its running state.

Figure 9.12. Using Services.msc to view and configure a Windows service properties.

[View full size image]


Writing Code to Create Connections

Okay, I'm finally ready to start talking about the code needed to open a connection. A big part of this discussion involves protecting the logon credentials and other ConnectionString details you use to gain access to the server. Microsoft seems very concerned with this aspect of security, but I'm not so frightened. That's because you must trust your developers and those that can review your codeif you can't, you're in trouble. In any respect, your DBA can protect your data and servers by setting up appropriate and specifically restricted Login accounts that can't do anything but what the application is intended to do. Visual Studio also has several new features that let you move your ConnectionString to the XML configuration file and encrypt it, if necessary. On the other hand, Peter has a good point:

I'm probably petrified. I trust the "authorized" developers and testers, but do I trust the users? Should I trust the users? Never, ever! Given, a user with a bit of technical background can determine the credentials being used in an database application and then use these credentials to get access to things they potentially should not is pretty scary. While your DBA might be happy for you to connect to the Accounting Database using the Accounting Client Application, they probably would be fired if they let you take a development tool like Visual Studio (or Access) and connected it to the Accounting Database. Yes, there are ways of limiting what a particular login account can do, but if the user doesn't know the credentials the application is using or the server that is being addressed by the application, they are at a greater and more controlled distance.

What Is a ConnectionString?

Okay, let's stop a second. I noticed there are a few newbies out there who might need a few terms defined before I go on. First, if you've worked with ADO classic before (or ADO.NET), you know that in order to open a connection, you need to point to a specific server instance and that SQL Server needs to know who you are (or who you're pretending to be). ADO.NET uses a ConnectionString to pass the answers to these questions to SQL Server. It's a bit early to talk about the specific contents of the ConnectionString, but I'll get to that later in this chapter. The ConnectionString is easy to code, but if you make a mistake coding it, your code can crash at runtimenot good. Consider using a class like the SqlConnectionStringBuilder to build the ConnectionString at design time. I'll show you how next.

Generally, your ConnectionString should not change during the life of your application. Each time it does, ADO.NET creates a new connection pool to manage it and all other connections with the same ConnectionStringthis can be very expensive. However, creating a different ConnectionString can help performance if you need a separate connection pool for some operations.

Building a ConnectionString with the SqlConnectionStringBuilder Class

If you need to create a ConnectionString programmatically, you can use the SqlConnectionStringBuilder class. It's certainly a better choice than concatenating key/value pair strings together and hoping the resulting string is a valid ConnectionString. That's because as you address the SqlConnectionStringBuilder object's strongly typed properties (or their synonyms), the class validates the entry on-the-flywell, for the most part. For example, (as shown in Figure 9.13), you can build up a valid ConnectionString to access SQL Server by selecting from a list of valid ConnectionString arguments.

Figure 9.13. Choose valid ConnectionString arguments from the SqlConnectionStringBuilder properties.

[View full size image]

I'm not using "late binding" to address the properties of the SqlConnectionStringBuilder object in the example shown in Figure 9.13. Although the documentation suggests this approach, it's problematic, as it delays resolution of the selected property until runtime and makes it easy to specify invalid keywords.

When you work with other .NET Data Providers (like System.Data.Odbc), you'll discover that several of the more important OdbcConnectionStringBuilder properties are not listed as properties. This might be an oversight or an intentional omission. In any case, you'll need to refer to these ConnectionString keywords using late binding. I illustrate that a bit later when discussing how to connect to flat files.

Tip

Yes, there are "ConnectionStringBuilder" classes for each ADO.NET provider. Each has its own properties that map to the provider-specific ConnectionString they build.

Figure 9.14 shows how to populate the SqlConnectionStringBuilder with property settings. As you enter each property, Visual Studio's Intellisense makes sure you choose from valid entries (at least, for most settings). Unlike other providers, SqlClient does not permit any "extraneous" entries in the ConnectionString, so be careful as you type late-bound entries that refer to keywords using Item("String of keyword") referencing.

Figure 9.14. Populating the SqlConnectionStringBuilder object with values.

[View full size image]

Protecting the ConnectionString

When you include sensitive credential information in a ConnectionString, it's important to protect those credentials from prying eyes. I'm of the opinion that it makes more sense to use credentials that permit access to just those parts of the database that the application is programmed to seeand no more. Sure, there are applications that need access to more sensitive information, and those will need to be granted access to those areas. In virtually all cases, it's important to protect the credentials. Fortunately, there are several strategies implemented in Visual Studio to help. First, you can set the BrowsableConnectionString property of the SqlConnectionStringBuilder object to false. This prevents the Visual Studio IDE from making the ConnectionString visible.

Persisting the ConnectionString

Visual Studio makes it easier than ever to save application or user "settings", which is a useful place to store a ConnectionString or any other value used to configure an application that you used to store in .INI files or the Windows Registry. These name/value pairs are stored behind the scenes in XML files, but fortunately, you won't have to manipulate these files directly. In a Windows Forms application, settings are placed in the app.config file. ASP.NET (and ASP Web Service) applications use the web.config file. There are two basic types of "settings":

  • Application settings: These are read-only settings that you initialize using the Solution Properties page, Settings tab (as shown in Figure 9.15). The name must conform to namespace requirement. The type when defining a ConnectionString should be "(Connection String)". The value (in this case) is set by clicking on the ellipsis (...) and filling in the Connection Properties dialog (as explained later in this chapter).

    Figure 9.15. Using the Solution Properties page to change the app.config file.

    [View full size image]

  • User settings: These are read-write settings (as shown in Figure 9.15) that can be initialized like application settings or in your code.

It's easy to address either type of settingbut don't bother trying to set application settings in code. Using the My.Settings.Save() method seems to work, but the values are not changed. For example, to fetch a saved ConnectionString from the app.config settings, you can use strongly typed referencing that exposes each of the settings as My.Settings propertiesthis means they'll show up in the Intellisense list of valid properties, as shown in Figure 9.16.

Figure 9.16. Getting and setting application and user settings in code using My.Settings.

[View full size image]

Building a Connection Object

I'm now ready to build an ADO.NET Connection object. For those of you new to .NET, remember that I'll use class New "constructors", which are a special class function whose job it is to initialize or "construct" an instance of a class. When the class is built using Visual Basic .NET or C#, you can include one or more "New" functions that help developers set up the instances of the class in the Visual Studio IDE. Data is passed to the New constructor as arguments, and these are used however the New function sees fit, but generally to set properties of the new class instance.

The SqlConnection object has two constructors, one that takes no arguments and simply returns an uninitialized SqlConnection object, and a second that accepts a properly constructed ConnectionString. That's the trickbuilding a string that has all of the required name/value pairs associated with a ConnectionString correctly spelled and with correctly scaled values. That is, if you provide a value for a named ConnectionString argument, it must conform to one of the valid keyword names, along with the correct datatype and range of permissible values for that keyword. That's where the table of acceptable values comes inunless you've used the SqlConnectionStringBuilder class to construct the ConnectionString (as shown in Figure 9.14).

Table 9.1 lists the valid names and aliases for keyword values within the ConnectionString. I've organized the keywords by function. Note that acceptable keywords must be spelled as shown (including the spaces) but are not case-sensitive. Do not confuse the SqlConnectionStringBuilder properties from the keywords they setthe properties are shown in (parenthesis).

Table 9.1. Valid ConnectionString Keywords and Aliases

Keyword/(Property)

Default

Description

Specifying Server, SQL Server Instance, and Database

AttachDBFilename Initial File Name Extended Properties (AttachDBFilename)

Attachable database (.MDF) file path (see discussion). Database file must be used with and match "Database" argument, which names the database being attached.

Data Source Server Address Addr Network Address (DataSource)

Addresses server either by NetBIOS name or by TCP address and port. Server=tcp:servername, portnumber. The default port is 1433 (see discussion).

User Instance (User Instance)

false

(SQLExpress only.) Creates a new private instance of SQL Server for the user. Copies the MDF file to the user's data area. Used with AttachDBFilename (see discussion).

Initial Catalog Database (InitialCatalog)

The name of the default database (see discussion).

Specifying Security

Integrated Security Trusted_Connection (Integrated Security)

false

If false, SQL Server User ID and Password must be supplied. If true (or SSPI), Windows credentials are used for authentication (see discussion).

Password Pwd (Password)

Recognized only when Integrated Security/Trusted_Connection is not used or is false. Indicates password for specified SQL Server Login ID.

User ID (UserID)

Recognized only when Integrated Security/Trusted_Connection is not used or is false. Indicates SQL Server Login ID. Requires matching Password argument (see discussion).

Encrypt (Encrypt)

false

If true and SQL Server is properly configured, SSL is used to transport all data sent to and from server.

Persist Security Info (PersistSecurityInfo)

false

When false, security information is not exposed on the Connection object (see discussion).

Specifying Options

Async[10] (AsynchronousProcessing)

false

If true, enables asynchronous operation support.

Application Name (ApplicationName)

.NET SqlClient Data Provider

Used to identify your application when using Profiler traces. Can be set to any string value.

Connect Timeout Connection Timeout (ConnectTimeout)

15

Number of seconds to wait for the connection process to complete. Not to be confused with CommandTimeout (see discussion).

Current Language (CurrentLanguage)

Language set on database

The SQL Server language to use.

Failover Partner | (FailoverPartner)

Names mirrored partner SQL Server. This keyword uses the same addressing scheme as Data Source (2.0 or later; see discussion).

MultipleActiveResultSets (MultipleActiveResultSets)

false

Enables multiple operations on a single connection (see discussion).

Network Library Net (Network Library)

dbmssocn

Network protocol. Can be dbnmpntw (Named Pipes), dbmsrpcn (Multiprotocol), dbmsadsn (Apple Talk), dbmsgnet (Via), dbmslpcn (shared memory), dbmsspxn (IPX/SPX), or dbmssocn (TCP/IP).

Packet Size (Packet Size)

8192

Network packet size in bytes.

Workstation ID (WorkstationID)

(local computer name)

Identifies workstation name. Can be used to filter Profiler traces. Can be set to any string value.

[10] This is documented as "asynch" (which does not work).

Before I move on, I need to pause a moment and more fully discuss a few of these keywords, as they're pretty important to your connection strategy. First, ADO.NET has tightened up their parser, so leading or trailing spaces and case are now ignored. This means "Server = Fred; Database = George" is the same as "Server=fred; Database=GeorGe". I don't recommend using values that contain a semicolon (;), but if that's necessary, you can frame the value in double quotes. Next, most Boolean values can be set to a number of alternative values. For example, true = yes and false=no in most cases. I suggest sticking with true or false. The only important exception is where Microsoft encourages us to set the Integrated Security keyword to "SSPI". While in parts of Texas and in the White House some versions of ADO.NET still accept "yup" in place of "yes", sadly, Microsoft dropped support of "you betcha" as a value setting some time ago.

Using AttachDBFilename

This keyword appeared in earlier versions of ADO.NET, but it's gained a great deal of flexibility with the innovations associated with (just the) SQL Server Express edition. Used alone, AttachDBFilename points to a SQL Server .MDF database file that's ready[11] to be attached to the master database. The AttachDBFilename value is set to the name of the primary database file, including the full path name of an attachable database. The path may be absolute or relative by using the DataDirectory substitution string. If DataDirectory is used, the database file must exist within a subdirectory of the directory pointed to by the substitution string. The DataDirectory is set by the application installer and is visible via My.Application.Deployment.DataDirectorybut not during debug. No, you can't put the .MDF on a remote server or provide an HTTP or UNC pathname to the .MDF file.

[11] By "ready," I mean it's been properly detached from another SQL Server instancenot just ripped out, leaving the wires hanging down.

When you use the AttachDBFilename keyword, ADO.NET opens the connection and checks to see if the named target database (as determined by the Database or Initial Catalog keyword value settings) has already been attached. If it's there, ADO.NET is done and you're granted access to the database (assuming you have permission). However, if the database has not been previously attached, ADO.NET permanently attaches the file to the target SQL Server instance. In this case, the "Database" or "Initial Catalog" keywords must name the database being attached. If it does not match, ADO.NET throws an exception and falls to the floor kicking and screaming.

Using AttachDBFilename with SQLExpress Instances

In the 2.0 Framework, ADO.NET dramatically expands the AttachDBFilename functionality. If you use AttachDBFilename, the keyword "User Instance" is set to true, and you're connecting to a SQL Server Express instance; the MDF database file and the SQL Server Express system database files (master, model, and tempdb) are copied to the user disk area under "Documents and Settings". Subsequently, this new private instance of SQL Server is started and a new process is spun up as the user who opened the connection and the MDF file is attached to the copy of master. Simple? Well, this approach means the user (based on the current Windows credentials) can have full use of the user database as well as the entire set of system databases without fear of impacting the "root" system database or any other user databases on the system.

Because the user has full access to a User Instance .MDF, it's possible to "safely" open the database connection as "SA" with full rightsbut since the user has full rights, there is no need to enable mixed-mode security. This AttachDBFilename approach was implemented because Microsoft wanted users to be able to run applications as usersand not have to have elevated administrator accounts simply because they were attaching a database deployed with their application. Sure, the same rules apply to the application itself as with other applicationsthey'll still have to be installed by the administrator.

Does this make sense for ASP.NET applications? Hardly. Remember, all ASP.NET applications run under the same process account by default; all of these applications would attach their local databases to the same shared instance of SQL Express. This also means the assurance that your data is safe because you're the only one using it is no longer possible. And no, you can't create a User Instance using any other version of SQL Serverit's a SQL Server Express feature.

Setting the Server Name Keyword

One of the challenges you'll face if you're unfamiliar with SQL Server instance addressing is how to code the "Server" ("Data Source" or "Address") ConnectionString keyword value. Consider that there are a number of ways to point to the SQL Server you wantjust remember to include the instance name to address. Yes, you can install SQL Server as the "default" instance, which is how preSQL Server 2000 installations were installed (there was only one instance). As discussed in Chapter 2, SQL Server 2000 introduced the concept of multiple instances, so when you install SQL Server 2000 on a system with an existing SQL Server 7.0, the 7.0 instance became the "default" instance (so existing code would not break). While it's possible to install SQL Server 2000 or 2005 as the default instance, it's not a particularly good ideaI recommend that you give each instance a unique name. Peter is a bit more flexible in this regard. He suggests that there are situations where installing default instances make sense. I'm not fully convinced, but he has more customers than I do.

To help visualize the addressing requirements, take a look at Figure 9.17. Here, I show two servers: machine "George" with a single "default" instance of SQL Server installed, and machine "Fred" with two SQL Server instances installed: "Test" and "Production". All of these machines are visible on the "Acme" domain. When creating a ConnectionString that must address machines across domains, include the domain name in the address you use in the "Server" keyword value.

Figure 9.17. Addressing servers and instances using the Data Source ConnectionString keyword.

[View full size image]

Let's review some sample Server/Data Source keyword value settings:

  • "<server name>" addresses the default SQL Server instance on the specified machine.

  • "." or "(local)" addresses the default SQL Server instance on the local machine.

  • ".\<instance>", as in ".\SS2K5", refers to the named SQL Server instance on the local machine.

  • "<domain>\<server>" addresses the default SQL Server instance on the named machine in a specific domain.

  • "<server name>\<instance>" addresses the named SQL Server instance on the specified current domain machine, as in "Fred\Test".

  • "<domain>\<server>\<instance>" addresses the named SQL Server instance on the named machine in the named domain, as in "Acme\Fred\Production".

  • "<IP>,<Port>" as in "209.20.233.22,1433" addresses default SQL Server instance on a machine-visible specific IP/Port.

Yes, you can also use the other aliases for the "Server" keyword (there are several), but I recommend use of the "Data Source" keyword and "DataSource" SqlConnectionStringBuilder property.

Establishing an Alias to Your Instance

All of the example applications included on the book's DVD reference the target SQL Server via an Alias of "DemoServer". An Alias permits you to address any named instance of SQL Server in the ConnectionString by using another namethus, the term "Alias". Creating an alias is very simplejust follow along:

1.
Start Microsoft SQL Server 2005 | Configuration Tools | SQL Server Configuration Manager. This MMC console application was discussed in detail in Chapter 2. It's used to manage the installed instances of SQL Server 2005, as well as establish aliases to any SQL Server instance.

2.
Click the "+" sign next to "SQL Server Native Client Configuration," as shown in Figure 9.18, to expose the Aliases tree. Right-click on Aliases to expose the operations menu and choose "New Alias..." to create a new Alias.

Figure 9.18. Opening the Aliases tree in SQL Server Configuration Manager.

3.
This opens the dialog shown in Figure 9.19, which prompts you for the Alias Name (in this case, set this to "DemoServer"; the Port No, which I left blank to use an automatically generated port number; the Protocol, which defaults to TCP/IP (which you can set as requiredjust make sure the SQL Server has the selected protocol enabled); and, finally, the Server entry. The Server entry requires that you name the server as you would in the ConnectionString. This can refer to a "local" instance of SQL Server or a remote server hosted on another system on your network.

Figure 9.19. Configuring a new Alias.

When you click "OK" in this dialog, the Alias is registered with the local system, so any references to the alias are translated to access the "Server" entry.

Revisiting Other Keywords

Let's take a closer look at some of the ConnectionString keywords to get a better understanding of how they interact with each other and tune how ADO.NET manages the Connection.

  • Connection Timeout: Generally, if things are working normally, SQL Server should be able to establish a connection in a couple of seconds or less. Of course, other factors can add time to that delay. For example, the first time you connect, it can take extra time to initialize the DLLS, create the pool, register your new connection in the pool, establish the connection, get the server-side agents started, and return status to your application. If SQL Server does not respond immediately to the data provider with a new connection handle (but does indicate that it's alive), the provider starts to wait Connection Timeout seconds. Setting the Connection Timeout does not always work as expected. When the network interface card (NIC) cannot see the Ethernet (as when the cable is disconnected or piece of the network infrastructure has melted down after that last lightening strike), the NIC timeout can add a great deal of time to the total time you'll wait before a timeout. Connection timeout exceptions are also very common when working with the connection pool, so I'll take up the discussion of that reason for timeout exceptions in a few pages.

  • Failover Partner: This keyword tells ADO.NET that, should the connection fail to open, it should try to connect to an explicitly named (failover) SQL Server setup to mirror the primary server. Of course, this means you'll need to have your DBA setup the mirrored failover SQL Server. Yes, it's possible to set up a failover partner SQL Server that's automatically referenced when the primary server fails. The Connection.DataSource property always reflects which server is actually in useassuming your connection is open.

  • Initial Catalog: This keyword names the "default" database used by T-SQL to resolve unspecific database object names. For example, if you code "SELECT Author FROM Pubs.dbo.Authors", the default database is not needed (it's stated explicitly), but if you coded "SELECT Author FROM dbo.Authors", the default database is assumed by the query compiler. The default database (or initial catalog) might point to "Biblio" or some other database, so the query might (very well) work differently. The default database is one of the few settings that can be changed after you open the connection by using the Connection.ChangeDatabase method. However, since the connection is reset each time it's opened (unless you disable this feature), the default database is changed back to its original setting when it's reopened. No, it's not a good idea to try to manage connection state (like current database) with the TSQL USE operatoralthough you can. That's because ADO.NET can't keep track of those settings.

    If you don't include the "Initial Catalog" or "Database" keyword, SQL Server uses the default database associated with the Login account named in the security credentials you supply. Of course, if the DBA changes this default value at some later time, applications that worked at one point in time might very well fail once the change is made. It's always a good idea to specify a default database (or Initial Catalog) keyword/value in your ConnectionString.

  • Integrated Security (or Trusted_Connection): This keyword tells ADO.NET to use the Windows login credentials used to launch your application as SQL Server credentials. That is, if a user logs into Windows as "Fred" in the "Acme" domain and runs an ADO.NET application that uses Integrated Security=SSPI[12], the application needs a Login account on SQL Server for Acme\Fred with rights to the default database. Yes, this is a lot of trouble, so you'll probably want to create a SQL Server Login that maps to a domain group that includes those individuals who need SQL Server access. As new people are added to the organization, you must remember to add them to the domain group so that they can get access to the data. Of course, as people leave, you need to remove them from the domain group.

    [12] Or Trusted_Connection=Yes, but Integrated Security=SSPI is preferred.

    Yes, if you're a member of the Administrators group, you're automatically granted rights to log on to SQL Server with SA (administrator) rights. No, don't assume that everyone logging on is a member of the Administrators groupexcept when using User Instances with SQL Server Express.

    In an ASP.NET application, IIS "owns" the connection identity, so when you use Integrated Security, ADO.NET uses the ASPNET (IIS 5.x) or IIS_WPG (IIS 6.x) Windows account (created when IIS is installed). In this case, you'll need to grant rights to the appropriate IIS accountgiving it permission to log on to SQL Server and grant appropriate read/read-write access to the initial catalog database.

  • MultipleActiveResultSets (MARS): This keyword tells ADO.NET to permit up to nine simultaneous SqlDataReader objects to share the same connection. That seems simple enough, but there are a number of restrictions that you should be aware of before arbitrarily enabling this feature. Note that "MARS" works only with SQL Server 2005 (or later) instances and has a number of other restrictions and side effects. See "Understanding MARS and its Implications," in this chapter.

  • Persist Security Info: If you need to prevent your code (or code you call) from seeing the security credentials in your ConnectionString, you need to set the "Persist Security Info" keyword value to true. Once set, you can fetch the ConnectionString from the SqlConnection object, but the security settings are stripped off the resulting string.

  • User ID and Password: If you don't set the "Integrated Security" keyword to SSPI or True, you'll need to provide a valid SQL Server Login ID and Password to gain access to a SQL Server instanceassuming (and this is a big assumption) that the instance has been configured to support mixed-mode security. This Login account must be granted rights to the default database, as well for the connection to be opened. By default, SQL Server is configured to accept (only) SSPI (and just SSPI) security credentials.

Configuring the Connection Pool Keyword Values

The next set of keywords, as shown in Table 9.2, is used to set up and manage the connection pool. Note that there is only one change in these arguments since ADO.NET 1.1: the "Connection Lifetime" keyword has a new alias, "Load Balance Timeout" (that Microsoft wants developers to use), that does the same thingwhich is nothing for ordinary (non-clustered) connection pools. These settings have corresponding properties in the SqlConnectionStringBuilder, as shown in (parenthesis).

Table 9.2. Connection Pool Keywords

Keyword

Default

Description

Connection Lifetime Load Balance Timeout (LoadBalanceTimeout)

0

(Clustered SQL Servers only.) Sets the number of minutes a closed connection remains in the pool. Has no effect on non-clustered connection pools. 0 = use default timeout (48 minutes).

Connection Reset (ConnectionReset)

true

When true, the pooler resets the live connection to its initial state when the connection is reopened (see discussion).

Enlist (Enlist)

true

When true, the pooler automatically enlists the connection in the creation thread's current transaction context. Since this incurs more overhead, I suggest setting this to false except when you're managing client-side transactions. See the BeginTransaction method.

Max Pool Size (MaxPoolSize)

100

The maximum number of connections allowed in a pool (see discussion).

Min Pool Size (MinPoolSize)

0

The minimum number of connections maintained in the pool (see discussion).

Pooling (Pooling)

True

Enables or disables connection pooling.

Don't place unvalidated strings into the ConnectionString (or any SQL), as this can lead to SQL injection attacks.


Understanding and Managing the Connection Pool

When connection pooling is enabled (and it is by default), the ADO.NET data provider initializes a sophisticated mechanism to manage connections to the .NET data provider. This mechanism is called a "connection pool". When your application code[13] executes the SqlConnection.Open method, the SqlClient provider creates a new connection pool, opens the connection to SQL Server, and places a handle to that connection in the connection pool. When the SqlConnection object's Close method is executed, in your code, the pooled connection is marked as "idle" but the actual server connection is left open. When another instance of your application (using the same ConnectionString) tries to open a connection, the pooler attempts to draw a connection from the pool of pre-opened, idle connections. If no idle connections are found, the pooling mechanism opens another new connection to the database and the pool grows in size. This process continues until the pool reaches 100 connections (a settable value) or the application/process that owns the connection pool endsin which case, the pool is discarded.

[13] Visual Studio also opens pooled connections as you work with the Server (or Database) Explorer or the other data tools.

This means when you close your database connection in code, you're really just releasing the connection back to a pool of used connections; the database connection to SQL Server remains open and continues to hold resources on the SQL Server instance until the pooler times out the connection and closes it. The idle timeout always takes between 4 and 8 minutesno, this is not a configurable number. However, if the same process using the same ConnectionString needs to reconnect, the connection pool simply plugs it back into an existing "idle" connectionassuming the transaction scope and other factors haven't changed. No, you might not get the same connection again, but in theory, the connection you get should be functionally equivalent to the original. This means that by pooling connections, your application can save the time that it would otherwise have to expend establishing a brand-new connection to the server.

As with any specialized functionality, connection pooling has a number of rules for its use. Here are some common questions you might have about the rules that govern connection pools.

  • How does one enable connection pooling? Connection pooling is enabled by default in all ADO.NET providers. You can disable pooling using the Pooling=False key/value pair in the ConnectionString. Don't disable pooling to correct problems with timeout exceptions.

    Tip

    Don't disable pooling in an attempt to correct problems with timeout exceptions.

  • When does the .NET Data Provider create a new pool? Your application might not always be able to use an existing pooled connection. If any of the following occurs, the SqlClient .NET Data Provider creates a new pool:

  • The ConnectionString property changes This property must be the same each time ADO.NET uses the Open method. Even if the only change is that the arguments are in a different order, the string is different. If you add, change, or remove argumentseven if they simply reflect the default behaviorthe property is different, and you get a new pool. The SqlConnectionStringBuilder can help make sure the ConnectionString is the same each time.

    • The User ID (UID) or password changes If you use a specific UID for each connection, each UID gets its own pool.

    • The process identifier (PID) changes Each process, program, component, Web Service, or web application domain gets its own pool. I discuss this aspect in a moment.

    • The transaction scope or enlistment changes If you're using transactions, each Transaction scope gets its own pool.

    • MARS enlistment changes If the connection uses multiple active resultsets (MARS), you get a unique pool.

  • Where are connection pools created? The provider always creates connection pools on the system where the code creating the connection runs: on the client. In a client/server application (for example, a Windows Forms application opens a connection to a remote server), the provider establishes the connection pool on the client (local system). In a Web Service, ASP, ASP.NET, or COM+ architecture, the code that opens the connection runs on a remote server that's hosting Microsoft IIS or Microsoft Transaction Server (MTS). In these cases, the "client" is the code running on the remote code server. Connection pools aren't created on the database server unless it happens to be hosting an application that opens a connection.

  • How does the security setting affect pooling? As I mentioned, if you use a specific UID for each connection, each UID gets its own pool. But if you specify Integrated Security=SSPI in the ConnectionString, you get one pool for your process's User Account. This means that you can't specify a role or UID and share the pool that the .NET Data Provider created for another role or user.

  • Why does the PID affect pooling? Like every database interface since the dawn of time, ADO.NET can't share connections across processes, so each process must have its own connection pool. A client/server application gets its own PID each time you run it. If you have five instances of an application, each gets its own PID, regardless of whether you run the application on one system or five systems. If you run a middle-tier componentsuch as a Web Service, a COM+ component, or a web applicationthat component also has its own PID; but in this case, multiple instances are running the same process. This means that a Web Service can serve a few clients or a few thousand clientsall of which share the same connection pool. When you're developing your Web Service, each time you compile and create a new assembly, the new executable gets a new PIDand a new connection pool. So, you could potentially have two nearly identical copies of an executable on your system, but each gets its own connection pool.

  • What happens when the pool is full? If the pool fills up with connections (the default limit is 100 connections), the .NET Data Provider waits for another instance of the application to close an existing connection and release it back to pool. If the .NET Data Provider can't get a connection in time (i.e., within the limit that the Connection Timeout setting specifies), it throws a System.InvalidOperationException with the message "Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached." You don't want to show this message to your users because it's not their fault and, generally, users can't debug your code (which caused the problem in the first place)[14] Note that ADO.NET uses System.InvalidOperation Exception for a variety of situations, so you need to parse the Exception Message string to find out what's really wrong.

    [14] When I get this message, I usually send a message to the web master offering to fix the problem.

  • When are pools destroyed? Connection pools are destroyed when the process that creates them ends. Because of this, an ADO.NET Data Provider running under IIS or MTS creates connection pools and destroys them when the process or Application Domain (AppDomain) ends. ADO.NET 2.0 also permits you to clear a specific pool or all pools associated with an application.

Tip

A bug exists in the Version 1.0 and 1.1 Framework performance counters that the SQL Server .NET Data Provider (System.Data.SqlClient) exposes. The counters falsely report the existence of active pools. To make sure that what you're seeing isn't a "ghost" pool, stop the Performance Monitor application, take a drink of coffee, and check again. When you're working with Windows Forms applications, pools drop off when the application or process ends. This behavior makes sense because the process creating the remote connection pool wouldn't necessarily end if IIS is acting as an agent for the web application or Web Service. This bug has been sidestepped in the 2.0 Frameworkthey replaced the counters with new ones. I'll discuss these next.

Be aware that if you shut down the SQL Server system or it shuts down on its own, any existing SqlClient connection pools remain in a zombie state unless IIS or MTS shuts down, too. In the 2.0 Framework, connection pools containing connections to dead or missing SQL Server instances are automatically flushed and rebuilt on next open.

Before I get any deeper into managing a connection pool, consider that 99% of this discussion is for ASP.NET applications that typically use dozens of pooled connections. A typical Windows Forms application will use one or two connections and really have little need for pooling.

Effectively Using Connection Pools

So what's the best way to use connection pools? Given the constraints that the .NET Data Provider implements, and if you aren't careful what valves you open and knobs you twist, you might think that it's easy to overflow the pool or server with connections. Remember that the limiting resources are the number of connections that a pool can handle, the number of connections available on the server, and the capacity of the server to meet the demand of these connections. As I noted earlier, the default number of connections a pool can handle is 100; when the number of connections within the pool reaches this value, the .NET Data Provider won't create more pooled connections. Ideally, your application should use a connection for a moment and release it back to the pool for other application instances to share, so 100 connections per pool should be more than enough.

Another tactic is to let the .NET Data Provider create any number of pools, again limited by the number of connections SQL Server allows according to your license or administrator settings. If some of your components are likely to compete for connections in a heavily used pool, it's a good idea to separate those components by creating two or more poolsone for each functional operation. One way to establish this separation is to set the application name in the ConnectionString to a unique value for each pool you want the .NET Data Provider to create. Watch out, though, because when a pool overflows (that is, you've exceed the set number of connections), your component performance grinds to a halt and your customers take their business to other sites.

Client/Server Pool Rules

Before you wade too deeply into how to manage your connection pool, you should reflect on what kind of application will benefit from connection management. Not all applications do. Client/server applications use pooled connections, but the specific instance of the client application is the only beneficiary. That's because each application runs in a unique process space, so each client application gets its own pool. Multiple copies or instances of the same Windows Forms application don't share the same pool even when they run on the same system; however, when a single application makes multiple connections (as many applications do), the .NET Data Provider manages all the connections in a common pool.

When you're working with client/server applications that need to manage the server-side DBMS more closely, the pool can be more of a hindrance than a help. In such cases, you might want the functionality of a like-new connection each time you use the Open method and an assurance that the connection actually closes when you use the Close method. Conversely, if the application repeatedly makes and breaks connections, the connection string remains unchanged, and you don't use MARS or transactions, the .NET Data Provider creates a dedicated pool for your connection. This means that the application reuses your pooled connection whenever possible, and your application can (potentially) run faster because the connection isn't being reopened and closed repeatedly. Remember, connection pooling does not make much sense for many client/server applications.

Wading into the Web

Over the last decade, the overwhelming focus at Microsoft has been web and middle-tier architectures, in which a code snippet's lifetime is shorter than a rabbit's heartbeat and replicated just as quickly. These routines typically have to open a database connection, execute a query, and return a response very quicklybefore the customer loses interest. Between one and N instances of the component simultaneously execute a processdozens to thousands of times per secondwith all the code snippets doing pretty much the same thing and using the same connection string.

If the component (such as a Web Service) takes too long to finish using the connection, the SqlClient Data Provider might add more connections to the component's private pool. Also remember that each .NET assembly gets a unique PID. This means that each new assembly you run gets its own poolregardless of the similarity of function or the fact that their ConnectionString values are identical.

Managing Lots of Swimmers

In any architecture, you need to carefully monitor three factors: the number of pools, the number of connections each pool manages, and the load on the server demanded by each connection. Fortunately, you can control all of these with code.

When your web site is under heavy use, a good strategy for making sure you have enough connections is to open each connection just before you need it and close it as soon as you canthe just-in-time (JIT) strategy. You can shorten the system time consumed by the connection by optimizing the query being executed and limiting the size of the returned rowset. ADO.NET can help because it opens and closes connections automatically (and quickly) when you use the DataSet Fill or Update methods. If you're using a SqlDataReader, you have to open and close the connection yourself. Even if you use the CommandBehavior.CloseConnection option, you still have to close the SqlDataReader to get the associated connection to close.

Unlike VB 6.0, the .NET languages can't guarantee your connection will be closed when a SqlConnection object falls out of scope. If the connection is still open, you (and the connection pool) can't reuse it. If the SqlConnection object is still open when it falls out of scope, it's probably lost forever. For example, the code segment shown in Figure 9.20 "leaks" a connection each time the GetData procedure executes because the SqlConnection isn't closed before the function that creates the SqlConnection object is completed. This leak happens because the code can no longer reference the SqlConnection object (which owns the pooled connection) after it falls from scope, and the .NET garbage collector won't necessarily clean up these orphaned objects (at least, not in time). The result? Connection timeout exceptions.

Figure 9.20. How not to handle DataReaders.

Let's try to fix some of the problems with this code. First, reconsider use of the SqlDataReader. If the application intends to bind to the rowset and provide post-fetch sorting, filtering, and finding, you probably should have created a DataTable instead. This is easier than ever in ADO.NET, as you can now load a DataTable object from a SqlDataReader.

But this solves only one problemclosing the SqlDataReader once rowset population is complete. Unless you set the CommandBehavior.CloseConnection option when the SqlDataReader.ExecuteReader method is executed, closing the SqlDataReader won't close the SqlConnection for us.

That solves two problems, but what if there is a problem when the query is executed? In this case, I need to close the SqlConnection to return it to the pool. Let's look at the modified code (shown in Figure 9.21) to see how it addresses these issues.

Figure 9.21. A better approach when passing DataReaders between routines.

When All Connections Are Gone

How your application behaves when all available connections are in use is crucial to the effective performance (and success) of your web site or application. A helpful tip is to remember that you might be able to make a temporary fix by increasing the value of the "Connection Timeout" connection-string argument or the SqlConnection object's ConnectionTimeout property. You use one or both of these values to set the length of time ADO.NET waits for a connection to be released from the pool and made available to your code. If you set the value too high, the browser (or client) might time out and throw a System.InvalidOperationException before you get connected. If you set the value too low, your error handler will have to know how to retry the connectionpossibly after asking the customer to be patient while your application deals with other requests. But increasing the ConnectionTimeout property is like moving the red line on a gauge that measures maximum performance to keep you from burning out a bearingit simply masks the problemright up to the point your server or engine melts down.

Activating, Tuning, and Disabling the Connection Pool

As I explained earlier in this chapter, when you're using the SqlClient .NET Data Provider to access SQL Server 7.0 and later, you need to create a new SqlConnection object and set its ConnectionString propertyit's the only way to change the SqlClient .NET Data Provider's behavior and control how it interacts with the connection pool. By using the appropriate ConnectionString keywords, you can turn off pooling, change the size of the pool, and tune the pool's operations.

Understanding Connection-Pool Options

Table 9.2 defines the SqlClient.SqlConnection.ConnectionString keywords that determine how ADO.NET[15] manages the connection pool for the specific connection you're opening. You can set these keywords on a connection-by-connection basis, which means that some Connection objects will be pooled and others won't be.

[15] No, other .NET Data Providers don't all have similar keywords to manage their connection pools.

  • Turning Off Connection Pooling: When you want more hands-on control of the server-side connection state or you want to debug a connection-pooling problem, using an unpooled connection makes sense, so it's nice to know that you can easily turn off connection pooling. If you're sure you want to turn off pooling, simply set the "Pooling" argument in the ConnectionString to False, and the SqlClient .NET Data Provider won't pool connections. In this case, the Open method establishes a new connection to the server (if one is available), and when you use the Close method, ADO.NET closes the server connection immediately. Be prepared for SQL Server to deny a connection-open request once you've exhausted the number of connections the SQL Server administrator allows or you exceed the number of connection licenses registered for the server. If your application is unable to connect to the server for some reason, your code needs to display a message to tell your user to come back later or simply wait and retry.

  • Setting the Maximum Pool Size: Use the "Max Pool Size" keyword to determine how many connections the pool manages. By default, the SqlClient .NET Data Provider permits 100 connections to accumulate in each pool. This doesn't mean that you start with 100 SqlConnection objects or 100 connections, only that you can't create more than 100 connections per pool. If you try to exceed the maximum limit you've set, your application waits until you wait for ConnectionTimeout seconds or until a connection becomes available in the pool. If time runs out, the pooler raises an exception. "Max Pool Size" doesn't limit the number of pools you can createthat limit is determined by the number of SQL Server connections that your SA setting or license makes available (about 32,768 per cluster).

  • Setting the Minimum Pool Size: In some cases, you might want to leave one or more connected SqlConnection objects in a pool for an indefinite length of time. If so, set the "Min Pool Size" keyword to an appropriate value. Setting this value can help connection performance for an application that doesn't run often but can't wait for the .NET Data Provider to construct a new SqlConnection object and establish a connection. You can also use this technique as a way to reserve connections. For example, you might set a "Min Pool Size" value if you want your Internet application to be able to deal with a flood of queries that grab all the connections from the pool but still ensure that you have a private connection. Or suppose you're hosting several applications on the same server and one of those applications is getting hammered with queries; by using this technique, you can reserve some connections for each application.

  • Cleaning Up the Pool: After you close a connection in code, ADO.NET notifies the connection pooler to release the connection back to the pool, and after a time (between 4 and 8 minutes), a connection-pool routine closes the database connection. No, the "release time" is not configurable. The "Connection Lifetime" ConnectionString keyword isn't what it appears to be. Microsoft included this option to handle a very special circumstancewhen you're using clustered servers. "Connection Lifetime" helps release connections more quickly when several SQL Servers are working in a cluster. In non-clustered systems, the argument has no bearing on how long the connection remains in the pool. After you close the ADO.NET SqlConnection object, connections remain open for the randomly selected period of time after your application releases them. The .NET Data Provider calculates the pool lifetime starting from the time the pooled connection was created. ADO.NET can also free the pools using new ClearPool methods available in the 2.0 version.

  • What About Other Providers? The way that ADO.NET interfaces with the connection-pool mechanism is newbut connection pools have been around for over a decade by now, so you should be fairly comfortable with how they work. However, if you're not using ADO.NET and the SqlClient .NET Data Provider, it's tougher to manage connection poolsmost providers don't have ConnectionString keywords to configure. If you don't understand the pooling mechanism, there's a good chance you'll create applications that work for a while but fail under stress. Or you might create applications that exhibit substandard performance, but you can't determine why. With the information in this chapter, you can explore how best to manage your connections by using the built-in pooling mechanism.

Monitoring the Connection Pool

So, you've opened and closed a connection, and now you want to know whether the connection is still in place. Let's look at some ways to determine how many connections are open and what they're doing, how many pools have been created, as well as how many connections are in each pool.

  • Run sp_who or sp_who2: These system stored procedures return a snapshot of information from the sysprocesses system table that shows the status of and information about all working processes. Generally, you'll see one server process ID (SPID) per connection. If you named your connection by using the Application Name argument in the connection string, your working connections are easy to find. Remember, SQL Server runs a number of SPIDs of its own, as does Visual Studio itself.

  • Use SQL Server Profiler: You can use the SQLProfiler TSQL_Replay template to trace open connections. If you're familiar with Profiler, this method is easier than polling by using sp_who.

  • Use the Windows Performance Monitor to monitor the pools and connections. I discuss this technique in a moment and walk through a set of examples in Appendix II, "Reinstalling the DACW and Other Missing Functionality in Visual Studio."

  • Monitor performance counters in code: You can monitor the health of your connection pool and the number of established connections by using hand-coded routines to extract performance counters exposed by the SqlClient provider. To make this easier, you can use .NET PerformanceCounter controls. The SqlClient performance counters in the 1.1 Framework were pretty buggyhopefully, the 2.0 versions (entirely new counters) won't be as problematic.

I show how to monitor SQL Server using the Profiler, the Performance tool, and Performance counters in Appendix III, "Monitoring SQL Server."

Keeping the Connection Pool from Overflowing

As you can tell from the preceding discussion, the connection pool keywords can dramatically impact the way applications (especially ASP.NET applications) perform and scale. Consider that simple (and well-written) Windows Forms applications do not materially benefit from the connection pool because they generally use only one or two connections at any point in time. However, in ASP.NET, applications in the connection pool permit ASP[16] page instances to open a connection quickly and return it to the pool for other instances of the page to reuse. Typically, in even a heavily used ASP application, the number of pooled connections is usually fairly low. Unless your server is pretty powerful, it might not be able to support more than a few dozen connections at once. The default setting of 100 pooled connections should be more than enough. If you know your ASP application is likely to build up a number of connections before it stabilizes, it's a great idea to set the "Min Pool Size" keyword to that number (or a few more). This means that when the first connection is opened, the pooler automatically opens N 1 more connections and leaves them open for the life of your application domain.

[16] By "ASP," I mean IIS-based applications that include ASP or ASP.NET architectures.

Sometimes, you can gain a bit of performance by turning off connection pooling in client/server applications, but this isn't advised for ASP appseven those that find the pool overflowing. As I discussed in earlier chapters, solving pool overflow problems is usually a matter of understanding how the pooler works. Remember, there are several typical scenarios that lead to pool overflow:

  • Unclosed connections: Make sure that any connection you open is closed. No, it's not necessary to use Dispose on the Connection object, but it doesn't hurt. Never depend on the CLR garbage collector (GC) to close your orphaned connections for you. While it might, it can take quite a while before the GC runs. Watch out for code that passes SqlDataReader to other classes or functions within your application. If not programmed correctly, this can easily lead to orphaned connections.

  • Exception handlers that bypass closing the connection. When something goes wrong as you process a query, the exception handler must be cognizant of the SqlConnection.State property. If the SqlConnection is left open for a common exception, the system will leak connections quicklyand slowly if the exception is irregularly encountered.

  • Connections closed by the server: When a stored procedure or other query terminates with a severity greater than 19, SQL Server closes the connection from its end. This can damage the pooled connection and make it unusable or orphan a connection if your code does not properly trap the exceptionand this might not be easy to do. Be sure to trap and deal with this (and all) SQL Server exception correctlyand close the connection to release the pooled connection. All data access code should be wrapped in a Try/Catch exception handler. In some situations, your only clue that the connection was lost is the fact that it's no longer usable.

  • Queries that overwhelm the server: If your application executes a query that takes too long to complete before another instance of your application starts a query that needs a connection, the pooler must create a new connectionit has no choice. If this trend continues, the pool will (perhaps slowly) overflow. If this is a concern, you need to monitor the pool using performance counters or the Performance Monitor application, as discussed in Appendix III.


Building a ConnectionString for Other Providers

Okay, so you need to connect to another data source and you need a ConnectionString. I don't have the bandwidth to cover every possible ConnectionString and the options associated with them, so I'll provide some typical ConnectionString settings to access the most common data sources[17]. Note that all of these data sources require you to choose another data access provider, as SqlClient is designed to connect to SQL Server versions 7.0 and later. Remember to:

[17] See www.connectionstrings.com for a comprehensive list of sample ConnectionStrings.

  • Choose a native .NET Data Provider like OracleClient whenever one is available. These will expose the most features and offer the best performance.

  • If a native .NET Data Provider is not available, try to find an ODBC driver and use the Odbc .NET Data Provider. This provider is light and fast, and is not COM-dependent.

  • At last resort, use the OLE DB interface via the OleDb .NET Data Provider. This is the slowest of all providers, as it requires use of COM interop to existing OLE DB Providers. Generally, you won't have to use OleDb unless you're accessing a JET/Access database.

Regardless of the provider you choose, you'll need to make sure your project includes a reference to the corresponding .NET Framework DLL that supports it. If you have experience with .NET, you'll have trouble finding some of the old references used in previous versions of Visual Studio .NET, as they've been consolidated into System.Data.

Accessing a Flat File

When you need to read a delimited text file, you'll want to use the "ODBC" .Text driver. Actually, it turns out that this driver actually launches JET to access the file (sigh). Whenever you need to access an ODBC data source, you have at least two choices: create an ODBC DSN or hard-code the needed ConnectionString keywords (what I call a DSN-less connection). Figure 9.22 shows how to open a delimited text file (a comma-delimited file, in this case) using ADO.NET. This approach (without the data binding) is an ideal way to create a SqlDataReader to pass to the SqlBulkCopy function so that the text file data can be quickly (very quickly) imported into a SQL Server table.

Figure 9.22. Accessing a flat text file.

[View full size image]

ADO Classic ADTG Recordset

If you use ADO classic to create a Recordset and persist the data with the Save method in ADTG format, you can read it using OleDb and the MSPersist provider. Note that the OleDb provider is also very particular about extraneous keywords passed in the ConnectionString. Figure 9.23 illustrates using the OleDb .NET data provider to read an .RS file created by the Recordset Save method. Note that the Command.CommandText is simply the path to the RS file.

Figure 9.23. Accessing ATDG persisted ADO classic Recordset (.RS) files.

[View full size image]

JET/Access Database

Getting to a JET database is a matter of resolving the path and using the JET OLE DB provider that matches the version of the .MDB being accessedunless the database has a User ID and Password assigned. In this case, you'll need to add "Database Password=<your password> and possibly "System Database=<path to system.mdw>".

The JET 4.0 provider knows how to access the current (4.0) version of JET databases but uses "installable ISAM" drivers to access older versions (which complicate deployment). I provide this information here to help import JET data into SQL Server tables. See Figure 9.24 for an example.

Figure 9.24. Accessing a JET database.

[View full size image]


Getting Visual Studio to Build Your ConnectionString

Note that Visual Studio .NET 2005 includes a new "Data Source" paradigm that can build a Visual Studio Data Source to point to any source of data, including databases, data "objects", and Web Services. I covered the Data Source Configuration Wizard in detail in Chapter 6, "Building Data Sources, DataSets, and TableAdapters," but in case you simply want to open a SqlConnection object against a specific database and aren't sure how to build the ConnectionString, you can simply create a new Data Connection in Visual Studio using the Server Explorer[18] and leverage the ConnectionString built by Visual Studio. The list of Data Connections exposed by the Server|Database Explorer are referenced throughout Visual Studio when referring to specific Data Sources. Note that creating a new Data Connection does not add any code to your project, but the Data Connections are persisted, so each time you start Visual Studio .NET, the Data Connections list is repopulated. Yes, referencing a Data Connection can add code to your application, as the ConnectionString it contains is leveraged to create SqlConnection objects behind the scenes.

[18] The "Server Explorer" is supported only in the "Professional" or better editions. In the "Standard" or "Academic" versions, some of the described functionality is exposed in the "Database Explorer."

The following steps walk you through the process of creating a new Visual Studio Data Connection.

1.
Click on the View | Server Explorer. This exposes the dockable Server Explorer tab, shown in Figure 9.25. Click on the icon shown to add a new Data Connection.

Figure 9.25. The Server Explorer tab.

Tip

Remember, we discussed the Server|Database explorers to manage T-SQL executables in more detail in Chapter 5, "Managing Executables with the Server Explorer."

2.
Click on the "Change..." button in the Add Connection dialog, as shown in Figure 9.26. Once you choose where the data is to be sourced, you can choose the appropriate .NET Data Provider from the drop-down list shown at the bottom of Figure 9.26. If you check the box "Always use this selection", Visual Studio defaults to the chosen Data source when new Data Connections are created. When you choose ".NET Framework Data Provider for SQL Server", you're choosing the SqlClient .NET Data Provider.

Figure 9.26. Change Data Source for new Data Connection.

3.
Once you choose a Data source, the Add Connection dialog morphs to the property requirements of the chosen provider. When working with SQL Server, the dialog captures the Windows server name that's hosting the instance of SQL Server you wish to access. You can also choose a server/instance name from the drop-down list (assuming the server and instance you need to access are visible on the Net). Figure 9.27 shows the list of visible servers and instances as populated. If the server is not visible, try clicking on the "Refresh" button. Be patient when clicking the down arrow or clicking "Refresh"the routines to locate SQL Server instances are not only slow, but they're lazy and don't always see available SQL Server instances.

Figure 9.27. Choosing a server and SQL Server instance name.

4.
Next, choose the type of authentication your connection expects to use. If you choose "Windows Authentication", the currently logged-on Windows user's (usually yours) credentials are passed to SQL Server to connect. If you have mixed mode authentication enabled and choose "Use SQL Server Authentication", you'll need to provide a valid Login ID (User name) and password. In the latter case, you can choose to expose these Login credentials in the ConnectionString by checking the "Save my password" check box.

5.
Once the authentication options are set, you can click the down arrow to choose a default database. If you don't choose a database name here, the default database is set to the default database set by the DBA when your Login ID was created. If your account is a member of the administrator's group, you'll likely be addressing the master database (not good). If the list does not populate or you get an exception when you click on the drop-down list down arrow, there is a problem with your credentials or the configuration of the server visibility (as discussed earlier in this chapter).

An alternative to choosing a database that's already managed by SQL Server is to point to an attachable .MDF database file. You can browse for that file using the "Browse..." button, as shown in Figure 9.27. The "logical name" is the name of the database contained in the .MDF file.

6.
Click the "Advanced..." button to open the Advanced Properties dialog, shown in Figure 9.28. This dialog can be used to view or change any of the ConnectionString properties. I recommend that you set the "Application Name" property to uniquely identify your application so it will be easier to find when using the Profiler. Note the entire ConnectionString is shown at the bottom of this dialog. If you right-click on this string and Select All, you can copy the ConnectionString to the clipboard and paste it into your application. Click "OK" to save your changes and return to the Add Connection dialog.

Figure 9.28. Inspecting and setting the ConnectionString properties with the Advanced Properties dialog.

7.
Click "OK" on the Add Connection dialog to create the new Data Connectionit should appear in the list of Data Connections in the Server Explorer. If the ConnectionString is the same as an existing Data Connection, Visual Studio ignores the "new" Data Connection.

8.
Once the Data Connection is created, you can right-click on the icon and click "Properties". This exposes the simple property page for the Data Connectionincluding the ConnectionString. This can be selected with Ctrl+A and copied to the clipboard with Ctrl+C.

The SqlConnectionStringBuilder is currently configured to set the MARS keyword to true. While Microsoft promised to fix this (the default is supposed to be false), you might want to double-check this feature before committing the ConnectionString.


Opening and Closing Connections (Semi-) Automatically

Once you have the ConnectionString configured, you would think that you're ready to open the Connectionyou aren't. Consider that ADO.NET manages the SqlConnection for you automatically if you use the SqlDataAdapter Fill or Update methods. In this case (as I'll discuss in detail in Chapter 10), if the SqlConnection is not yet open, the Fill or Update methods open the SqlConnection (associated with the SqlDataAdapter), run the query, return the rowset(s), and close the SqlConnectionall in one atomic step. If the SqlConnection is already open when Fill or Update is executed, the SqlConnection is left open after the methods are done. If something goes wrong during the Fill or Update operations, ADO.NET ensures that the SqlConnection is returned to its initial state.

If you intend to build a SqlCommand and create a SqlDataReader, you'll need to execute the SqlConnection Open method yourself. Once you take on this responsibility, you also carry the burden of making sure the SqlConnection is closed once you've completed reading all of the resultsets from the SqlDataReader. As discussed in Chapter 10, you can get the SqlCommand.ExecuteReader method to close the SqlConnection for you when (and only when) the SqlDataReader is closedeither with the SqlDataReader Close method or when binding to a complex bound control like a DataGridView.


Understanding the Connection Properties

The SqlConnection class is derived from the System.Data.Common.DbConnection class. No, I have not talked about the Data.Common namespace, as I plan to touch on this class in an upcoming EBook. Suffice it to say that, like most .NET Framework classes, SqlConnection inherits a number of properties common to all .NET Data Provider "Connection" classes.

Table 9.3 shows the key SqlClient properties that you're likely to useI'll leave the details of the .NET object-management properties to the Microsoft reference topics. Many of these properties are set through the ConnectionString or the SqlConnectionStringBuilder, so I've already discussed them quite a bit.

Table 9.3. Key SqlConnection Properties

Property

Data Type (Default)

Description

ConnectionString

String

Addresses server, specifies credentials, and configures Connection object.

ConnectionTimeout

Integer (15)

Number of seconds to wait before abandoning connection attempt.

Database

String

Initial catalog or default database.

DataSource

String

Addresses SQL Server instance.

FireInfoMessageEvent OnUserErrors

Boolean (False)

When set to true, errors previously treated as exceptions are handled as InfoMessage events (see discussion).

PacketSize

Integer (8192)

Determines size in bytes of network packets used to communicate with SQL Server. Larger packets mean fewer packets but (potentially) more wasted space.

ServerVersion

String

Returns SQL Server version hosting connection.

State

Data. ConnectionState

Returns state of connection. Open, Closed, Connecting, Executing, Fetching, Broken (not used).

StatisticsEnabled

Boolean (false)

Enables gathering statistics for connection (see discussion). Use the RetrieveStatistics method to retrieve the performance statistics.

WorkstationID

String (name of client computer)

Names the database client.

The FireInfoMessageEventOnUserErrors property is new for the 2.0 Framework. It addresses a problem voiced by stored procedure developers. They found it easier to handle RAISERROR messages in the InfoMessage event handler instead of having lower-severity errors appear in InfoMessage and higher-severity errors returned by (expensive) Exception Catch routines. When set to true, all events fire immediately and are handled by the event handler. When set to false, InfoMessage events are handled at the end of the procedure.

The StatisticsEnabled property is also new for the 2.0 Framework. Apparently, enabling statistics gathering has a minor but measurable (negative) effect on performance and, therefore, should be enabled only when it is required. This property exposes another way to provide real-time monitoring of the SQL Server operations used to execute your query. These connection statistics are reset when you close the connection (as it's returned to the connection pool). Use the RetrieveStatistics method to retrieve the performance statistics and ResetStatistics to reset the current set of statistics, as shown in Figure 9.29. Note that individual metrics can be addressed by name from the IDictionary object created by the RetrieveStatistics method.

Figure 9.29. Capturing performance statistics from a SQL Server query.

[View full size image]


Using the Connection Methods

There are a host of methods on the SqlConnection class and several new ones implemented in the 2.0 Framework. I've already talked about some of these, including the Open and Close methods, and how you don't necessarily have to use these in your applications (if you use the SqlDataAdapter Fill or Update methods). Table 9.4 lists the key SqlConnection methods. I'll leave out those that you won't be likely to need and focus on those you'll use or abuse.

Table 9.4. Key SqlConnection Methods

Method

DataType (Default)

Description

ChangeDatabase

String

Requests initial catalog/default database to be changed (see discussion).

ChangePassword

String, String

New for 2.0 Framework. Permits ADO.NET to change a Login account's password (see discussion).

ClearAllPools

(none)

New for 2.0 Framework. Clears all connection pools of idle connections owned by this process (see discussion).

ClearPool

SqlConnection

New for 2.0 Framework. Clears specific pool of idle connections (see discussion).

Close

(none)

Set a pooled connection as idle or closes an unpooled physical connection (see discussion).

Dispose

(none)

Closes the connection and releases its resources (see discussion).

New

(none)

Creates a new (empty) instance of the SqlConnection class. You'll have to fill in the properties by hand before the Connection can be used.

New (constructor)

String

Creates a new instance of the SqlConnection class using a ConnectionString to initialize the instance (see discussion).

Open

(none)

Attempts to open a physical connection to the Data Source specified in the SqlConnection instance properties (see discussion).

ResetStatistics

(none)

If StatisticsEnabled is true, this method flushes the current set of operational statistics. (See the StatisticsEnabled property.)

BeginTransaction

(various)

Starts and returns an ADO.NET-managed Transaction-based on the arguments passed in the constructors (see discussion).

CreateCommand

(none)

Creates an empty SqlCommand instance associated with the connection (rarely used).

GetSchema

(various)

New for 2.0 Framework. Returns schema information from the Data Source as a DataTable (see discussion).

RetrieveStatistics

(none)

Returns an IDictionary set of name/value pairs containing the Connection operational metrics (see the StatisticsEnabled property).

Let's take a closer look at some of the more important methods.

  • ChangeDatabase: This method requests that ADO.NET change the initial catalog (database) or default database setting. Of course, this assumes that your credentials have rights on the new default database. If you don't or the string supplied is not a valid database name, ADO.NET throws an exception. This change (if it succeeds) is temporary. That is, when you close the connection and it's returned to the pool, the original connection string setting is appliedeven if there is no value set for the Initial Catalog/Database keyword. In this case, the initial catalog is set based on the default database assigned to the Login account. This means if your application reopens the connection, the initial catalog is the same as when the Connection is first opened.

  • ChangePassword: When the SQL Server Login account password expires, your application will trip a SqlException with the Number property set to 18487 (password expired) or 18488 (password must be reset). The new 2.0 ChangePassword method permits you to change the password. The first parameter must be a valid ConnectionString using the Login name that failed and the existing (expired) password, along with enough information to address the Data Source. The second parameter must be a valid password that conforms to whatever password restrictions are set by the SQL Server configuration.

  • ClearAllPools: This new 2.0 method (of the SqlConnection class) closes the physical connection of all idle connections in all pools created by the current process. Those connections in the pool that are still open are marked for permanent closure, so once you execute Close (or Dispose), these connections are closed.

    The pool flusher methods are useful in situations where you need to ensure that no connections are being held in the pool while you perform administrative work on the serverfor example, when detaching databases or attempting to switch SQL Server into "single-user" mode as required by the recovery routines. No, don't expect to hear any noise while these methods executethey don't make a flushing sound except in parts of Australia where the water spins in the opposite direction.

  • ClearPool: This new 2.0 method (of the SqlConnection class) closes the physical connection of all idle connections in a specific pool, as referenced by the SqlConnection object. Those connections in the pool that are still open are marked for permanent closure, so once you execute Close (or Dispose), these connections are closed. The following example (see Figure 9.30)[19] shows the code to clear the connection pool for a specific SqlConnection object (cn) and all pools for the current process.

    [19] See the "Connection Methods" example on the DVD.

    Figure 9.30. Flushing the connection pools.

    [View full size image]

  • Close: This method sets a pooled connection to "idle" and changes the State to Closed. If applied to unpooled SqlConnection objects, the physical connection is closed. Closing a connection rolls back any pending transactions. Never depend on an object's Finalizer to close connections for you.

  • Dispose: The Dispose method calls the Close method, releases all resources held by the Connection, and resets its propertiesit's not necessary to call both Close and Dispose. Don't use Dispose if you plan to reuse the SqlConnection object.

  • New (Constructor): The New method creates a new instance of the SqlConnection class. If you pass a string argument, it's considered to be a valid ConnectionString. The constructor code parses the ConnectionString and populates the Connection properties. If an invalid argument is supplied, a System.ArgumentException is thrown.

  • Open: This method takes the properties set in the ConnectionString property and attempts to open a connection to the specified data source using the credentials and option settings provided. I've discussed this at length throughout this chapter. Once you open the connection, the ConnectionString property is frozen. If you set "Persist Security Information" to true in the ConnectionStrin any access to the ConnectionString property is returned stripped of its security credentials. Most of the other properties are also frozen once the connection has been opened at least once.

  • BeginTransaction: This method initiates a SQL Server Transaction and marks the Connection as participating in a transaction context. Once a SqlConnection is enlisted in a Transaction, it cannot be reused, so an independent pool is created to manage it. Closing a SqlConnection automatically rolls back any pending transaction.

  • Caution: If you initiate a Transaction and a Severity 16 (or higher) error occurs, the transaction is not automatically rolled backthat's up to you.

  • GetSchema: This new method for the 2.0 Framework permits tool developers to retrieve data structures (schema) information about the data source, database, table, columns, or any specific object in the database. I'll be focusing on SQL Server here, but this technology is included in all of the providers, so OLE DB developers can use it to mine the schema of their JET databases. While coding the GetSchema method is very simple (as shown in Figure 9.31), the voluminous data it returns seems somewhat daunting at first.

    Figure 9.31. Using GetSchema to fetch a list of databases visible on the Connection.

I've created a simple example application[20] to view and filter the metadata returned by the GetSchema method. I expect that simply running this application will lead to a much better understanding of how GetSchema works.

[20] See "GetSchemaExample" on the DVD.

GetSchema works against a specific SQL Server connection and a default database. You can point to another database (as I do in the example application) by using the Connection.ChangeDatabase method. GetSchema returns a DataTable containing a set of rows that contains metadata from the selected MetaDataCollection elements. These entries are shown in Table 9.5.

Table 9.5. GetSchema Collections and What They Contain

Collection Name

Description

DataSourceInformation

Returns metrics regarding SQL Server and how to interface with it.

DataTypes

Lists datatypes and their metrics, including column size, parameters need to create the type, and more.

Restrictions

Lists the server-dictated restrictions on access to the listed elements.

ReservedWords

Lists the strings used in T-SQL and elsewhere that may not be used as identifiers, like database, table, view, stored procedure, or column names.

Users

Lists all user names.

Databases

Lists all databases

Tables

Lists all tables by database. Note only the default database is listed.

Columns

Lists all data table columns, including all column metrics.

Views

Lists all views and its metrics.

ViewColumns

Lists all columns of all views.

ProcedureParameters

Lists all parameters for all data source stored procedures.

ForeignKeys

Lists all foreign keys and their metrics.

IndexColumns

Lists all Columns from all Indexes and their metrics.

Indexes

Lists all Indexes and their metrics.

UserDefinedTypes

Lists all CLR User-defined datatypes.

Since GetSchema returns a DataTable that includes all columns or parameters from the entire database, you'll want to set limits to focus on just those objects you need to examine. To this end, ADO.NET exposes "Restrictions" that are used to limit the amount of metadata returned from GetSchema. These Restrictions are implemented by passing an array of strings to the GetSchema method (as shown in Figure 9.32). The array must map to the Restrictions metadata returned for each metadata object. Yes, this is returned by the same methodGetSchema ("Restrictions") returns a DataTable with all of the Restrictions for all of the objects (that have restrictions). Figure 9.34 shows a DataGridView populated from a bound DataTable containing the valid Restrictions for a selected metadata element (in this case, ProcedureParameters).

Figure 9.32. A DataGridView control bound to a DataTable containing valid Restrictions.

[View full size image]

Figure 9.33 shows how I take the contents of a DataTable containing all of the Restrictions (that's been bound to the DataGridView) for the specific object collection and populate the string array of Restrictions. This array is passed to GetSchema along with the current metadata element to return a filtered list of elements. Yes, it's possible to set any number of the Restriction "values". Note that the Value must be set to Nothing (or null in C#) if you do not want that value to be tested.

Figure 9.33. Populating the Restrictions string array from a DataGridView.

[View full size image]


Handling Connection Events

The SqlConnection object exposes a few events that can be useful to trap "back-channel" communications from SQL Server. There are no new events implemented by ADO.NET 2.0, so I'll have to make do with those pulled forward from ADO.NET 1.1. Handling the Connection events is simple for Visual Basic .NET developerssimply declare the Connection object using the WithEvents operand. In C#, you'll have to set up a delegate to the SqlInfoMessageEventHandler.

The Connection events are the:

  • Disposed event: This event fires when the object is disposed. There is little information about what should or should not happen here.

  • InfoMessage event: This event is used to return an "informational" message back to your application. For example, if the T-SQL being executed includes a PRINT statement, the text message is returned via the InfoMessage event handler. RAISERROR also uses this mechanism to return messages if the severity is 10 or less. Note that in ADO.NET 2.0, a new Connection property, FireInfoMessageEventOnUserErrors, has been added. When you set this property to true, errors that previously were treated as exceptions are handled as InfoMessage events. All events fire immediately and are handled by the event handler. If is FireInfoMessageEventOnUserErrors is set to false, InfoMessage events are handled at the end of the procedure.

  • StateChange event: This event is fired when the Connection object's State property changes. No, even though there is a Broken state, this is not implemented in ADO.NET 2.0it's simply a placeholder.

The example shown in Figure 9.34 shows how to set up the Connection event handlers in Visual Basic .NET and implement code to deal with the returned values. Because the FireInfoMessageEventOnUserErrors property has not been set to true, the RAISERROR messages generated by the T-SQL are not passed to the InfoMessage event handler.

Figure 9.34. Using WithEvents to set up Connection Exception handlers.

[View full size image]

Sometimes, your application design requires that you leave the SqlConnection open for the life of the applicationor at least for extended periods of time. In other designs (especially ASP.NET and Web Services), you'll be prevented from holding SqlConnection objects open. In this case, you need to avoid the most fundamental of mistakesleaving the SqlConnection in an open state when it falls from scope. I've talked about this a number of times, so I won't pound on this further. However, I do need to briefly touch on an old saw about using Connection.Dispose instead of or in addition to Connection.Close. OO purists will say, "If the class implements a Dispose method, you should call it when you're done with the object." I say to them in response: "Who's going to pay for all that extra (and unneeded code)? Are you aware that the TextBox, Label, and every other control has a Dispose method? How many programs do you see that call Dispose on every single one of these controls. Let's not go there."


Managing Connection Exceptions

One of the most sophisticated parts of any serious application is the exception handlers. Since connecting to any external device is fraught with potential issues, your code must often set up a "state machine" to decide what to do. In any case, any production code should include a TRy/Catch exception handler to trap the exceptions that can occur.

One thing you shouldn't do is pass the responsibility of handling exceptions to your customersthe end-user. These folks are the least likely to be able to help the situation, so presenting them with an ADO.NET exception message just because your server is bogged down is just laziness and unprofessional.

The complexity of getting connected and having to deal with the issues yourself does not make your job any easier. This means you're going to have to:

  • Determine the current application state. That is, what were you trying to do when the exception occurred? If you're in the process of opening your first connection, the list of potential problems is far longer than if this is the 99th time you've connected.

  • Determine whether you have connectivity. Is the network card, the network, and all of the Ethernet (or whatever) hardware between you and the server working?

  • Determine whether the server hosting SQL Server is up and visible. Has the server been taken down for maintenance or stolen in the night? Is it overloaded and unable to accept more work?

  • Determine whether the SQL Server instance you're addressing is installed and running. Are you accessing the primary or failover server? Remember, you must include an instance name in the Data Source/Server keyword if you aren't addressing the default instance.

  • Determine whether the SQL Server login accounts are set up to permit you to access the server and initial catalog database. While members of the Administrators Windows group have rights to access the server as SA, members of the Users group or other groups don't have rights by default. If you're using ASP applications, have you correctly configured the ASPNET account?

  • Determine whether the Login credentials are currentyour password might have expired and you might need to use the ChangePassword method.

  • Determine whether the SQL Server instance is configured for SSPI or "mixed" SQL Server security. In any case, you still need to set up login accounts for the credentials you're using, unless you're a member of the Administrators group.

  • Determine whether this is first time the exception has occurredthat is, have you already retried the operation? If your connection fails because of a bad connection in the pool, you might need to flush the pool before retrying the connection.

  • Determine whether the exception is caused by a full or overloaded connection pool. I've just spent a dozen pages discussing this possibility. You know what to do to fix the problem.

  • Determine whether the file you're trying to attach is in an "attachable" state. Simply copying an .MDB file from one server to another does not mean the file can be attached. Make sure the .MDB file has been properly detached from the source server before trying to install it elsewhere.

  • Determine whether the firewall is preventing access to the SQL Server ports TCP 1433 and UDP 1434, and port 49 for Named Pipes. Is SQL Server configured to use these ports and the needed protocols? Remember, setup disables these ports and the external protocols by default.

  • Determine the load on the server. Have you exhausted the number of available resources on the server? Remember that SQL Server can be configured to limit the number of inbound connections. If the host server or SQL Server is too busy, it might take (far) longer to connect. Is the host server being used to host Reporting Services, Indexing Services, or a print server? These services can consume a great deal of processing resources and memoryyou might have consumed all that the server has available.

  • Determine whether you're using MARS. If you've left MultipleActiveResultSets enabled, you'll discover a number of obscure exception conditionssome of which occur after only nine connections have been made.

Sourcing Exceptions

When connecting, you're generally going to experience at least two types of exceptions. While I assume you know how to write basic exception handlers, let's review a few points. First, I won't discuss "unmanaged" error handlers, which are provided in Visual Basic .NET (but not C#) for backward-compatibility with Visual Basic 6.0.

Exception handlers consist of a try statement followed by the code to monitor for exceptions, one or more Catch statements that mark the end of the monitored code block, and the beginning of the "exception handler" Catch block(s). The Catch statement specifies the Exception class to trap. Remember that the Exception class is the "base" class and should trap all exceptionswell, almost. After a while, you'll discover that if you use Windows controls to handle data, these controls can throw exceptions as well, and your exception handler won't be able to trap them. After the Catch blocks, you can add an optional Finally block that's executed whether or not there is an exception to catch. This is a great place to put the Close method for the SqlConnection to make sure it's closedwhether or not there is an exception during its use.

  • SqlException: After the ConnectionString has been accepted and ADO.NET attempts to open the SqlConnection, you can expect a SqlException to fire if the server is not running, if the server is not configured correctly, or if some other server-related issue occurs. I examine this class more closely later in this section.

  • (General) Exception: If you pass invalid settings to the SqlConnectionStringBuilder, or when declaring the SqlConnection object with a fleshed-out ConnectionString, ADO.NET throws a general Exception. Your Exception handler Catch routine needs to trap more specific exceptions first (as shown in Figure 9.35). Remember that (virtually) all exceptions are caught by the Catch Exception routine.

    Figure 9.35. Using multiple Catch statements to trap SqlExceptions.

    [View full size image]

Understanding the SqlException Class

The SqlClient .NET Data Provider is responsible for implementing its own "exception" class based on Exception. This implementation exposes SQL Server-specific information about went wrong. For example, a SqlException exposes the Errors collection of SqlError objects that provide even finer granularity to SQL Server exception reporting. Table 9.6 describes the SqlException properties.

Table 9.6. SqlException and SqlError Properties

SqlException Public Properties

SqlError

Type

Description

Class

Byte

Severity. A value from 1 to 25, as set by SQL Server or RAISERROR.

Data

System.Collections. IDictionary

Inherited from Exception. A collection of key/value pairs that provide additional, user-defined information about the exception.

ErrorCode

Integer

Returns the HRESULT of an error.

Errors

SqlClient. SqlErrorCollection

Returns a collection of SqlError objects (see discussion).

HelpLink

String

Inherited from Exception. Links to help on this error.

InnerException

Exception

Links to Exception generated by called routine.

LineNumber

Integer

T-SQL query, stored procedure, or batch line number that generated error.

Message

String

Inherited from Exception. Describes error.

Number

Integer

A number that identifies the type of error.

Procedure

String

Names stored procedure or RPC that generated error.

Server

String

Names Data Source server that generated error.

Source

String

Names provider that generated error (System.Data.SqlClient).

StackTrace

String

A string containing the call stack leading up to the error.

State

String

An arbitrary code set by RAISERROR or SQL Server indicating type of error.

TargetSite

System.Reflection.MethodBase

Inherited from Exception. Returns method that threw exception.

Note that the Errors property returns a collection of SqlError objects. The SqlError class also contains a number of useful properties that are a subset of the SqlException properties, as indicated in Table 9.3. The example shown in Figure 9.36 illustrates a (very) simple SqlException handler to dump the returned error information to the debug window. When this code runs, the PRINT message is suppressed because of the RAISERROR executions. The SqlError messages are returned in "reverse" or "inside-out" order. That is, the "high-severity" message is returned first, followed by the "low-severity" message.

Figure 9.36. Handling nested SqlException and SqlError error messages.

[View full size image]

Dealing with Exceptions

Okay, something went wrong. What can/should you do about it? I can offer a few suggestions:

  • Trap it. Each and every application you write needs to ensure that all exceptions are trapped and handled one way or another. You do not want your users to be frightened out of their wits by scary .NET exception dialogs.

  • Report it. If the exception (even a minor exception) is logged, you'll be able to know how often it occurs and get a better handle on how to discover what's causing it. If it occurs each morning between 8 and 8:15, it might be the time the clerk plugs in the coffee maker into the outlet shared by the server. I often record exceptions to the Windows events log or a custom log file maintained for the application.

  • Pass it up. You might find it necessary and expedient to simply pass the exception to the calling routine. You can set the InnerException of a new Exception that you fire.

  • Retry it. If you're dealing with a transitory issue, waiting a bit and retrying might be your best choice. The problem is that unless you do some exploring, you won't know the source of the problem or whether it's worth waiting for it to resolve itself.

  • Root it out. Connectivity issues are hard to diagnose, but there are a number of ways to isolate the problem. Try to connect on the host server with SQLCMD using the credentials your application is using. If you're using SSPI connectivity, the credentials used on the host server might not be the same as those being used by the client system. See if you can ping the server and use Telnet to probe the ports. See if other applications like SQLCMD can access the server. Can the SQL Server tools like SQL Server Management Studio or SQL Express Manager connect to the server?

  • Repair it. If your connection string is incorrect or you haven't set up the accounts correctly, you'll need to correct these settings before retrying. However, if the accounts that were working before are no longer working, you need to have a talk with the DBA. If you're the DBA, it's still your fault unless you gave away the SA password (then it's still your fault). Do some hardware tracing to see if you have connectivity. Watch for changes in software or hardware firewalls.

  • Ignore it. Yes, there are times when it makes perfect sense to ignore an exception and move on. As you progress through the chapters, you'll find many situations where stored procedures return low-severity messages that can be ignored or simply logged.

  • Avoid it. It's an even better practice to code your application to prevent needless exceptions. Don't use Exception handlers as a back-channel communications methodthat's what the InfoMessage event is forit's simply too expensive.


Summary

This (long) chapter dissected the SqlConnection object and the mechanisms ADO.NET and the SqlClient .NET Data Provider implement to manage, pool, and dismantle connections. So many features are enabled, hidden, exposed, or disabled by use of the SqlConnection properties (especially the ConnectionString) that we had to spend a long time working through this content. Once you're connected to SQL Serverregardless of the versionyou're ready to start executing queries. Remember, the SqlConnection object is also the place where InfoMessage events are returned, as well as some SQL Server errorsso be ready. Next stop is the SqlCommand object.


Chapter 10. Managing SqlCommand Objects

    Introduction

    Creating SqlCommand Objects

    Integrating Ad Hoc Queries into Your Application

    Coding Parameter Queries

    Validating the Value

    Executing Stored Procedures

    Summary


Introduction

Once your application is connected to SQL Server, you should be ready to start making intelligent use of the server. This means asking intelligent questions that return intelligent answers. Remember that SQL Server does not make a particularly good "file" server, although too many developers treat it as such. Database table data belongs on the servernot in client memory, where the data gets stale faster than cheap Swiss cheese drawing flies on the kitchen counter. Unfortunately, too many examples are dumbed down and written to "simplify" the process of making queriesyou and I often see "SELECT * FROM Customers" as the example query. Too many drag-and-drop demonstrations show how easy it is to take a data table or two and generate a strongly typed class (with relationships) in a matter of seconds. That makes a cool demo, but unless you take the extra steps to parameterize these queries, you'll end up with an unscalable application that's slower than a Volkswagen bug towing a 65-foot boat trailer up Mt. Rainer.

As I lead you through this chapter, I'll point out security risksdark alleys and tough neighborhoods that can ravage your data and batter your career. I'll also make it clear the routes that yield the best performance and productivity. As a result, your application should be faster, safer, and more scalable than your competition's.

In a nutshell, this chapter shows you the mechanics of building ADO.NET SqlCommand objects and a Parameters collection, and how to use these to execute ad-hoc queries, stored procedures, maintenance queries, or whatever you want to execute. Remember that SQL Server is tuned to run most efficiently when you execute stored procedures, so I'll spend quite a bit of time talking about the special issues you'll face when executing stored proceduresthings like multiple resultsets, OUTPUT parameters, and handling Parameter collections. I'll also talk about using the new SqlBulkCopy class to quickly move data from any data source to SQL Serverthat's because it makes more sense than using any SqlCommand approach.

ADO.NET Command objects play a crucial role when you choose to use the SqlDataAdapter or TableAdapter classes. In this chapter, I'll also discuss how to leverage these classes using Visual Studiogenerated code and custom hand-rolled code. I'll provide enough detail so you can decide which makes more sense for your development needs.


Creating SqlCommand Objects

If you're new to ADO.NET but familiar with ADO.NET classic, you might not realize that there is no way to execute a query (or any T-SQL command) without using a SqlCommand object of some kind. The SQL Server SqlClient.NET Data Provider exposes these as the SqlCommand class. In ADO.NET classic, you can simply execute SQL right off the SqlConnection object, by using the Recordset Open method or several other techniques.

This is not the case with ADO.NETyou must create a SqlCommand object and populate its properties to execute the SQL. Yes, there is a CreateCommand method on the SqlConnection class, but it has no overloads, so it does not save you any time, money, or wear and tear on your keyboard. However, I don't know how much easier it can be to write a couple of lines of code to create and configure a simple SqlCommand object, as shown in Figure 10.1. Sure, you're going to have to create a SqlConnection object to link to the SqlCommand, but that's easy, too.

Figure 10.1. Instantiating and configuring a SqlCommand object.

[View full size image]

Understanding the SqlCommand Properties

The SqlCommand class exposes a number of basic properties that help ADO.NET understand how to execute it. These contain the SQL query, the type of query, and how long SQL Server should take to execute it (before giving up). Let's take a close look at these properties, as shown in Table 10.1each of these properties is discussed in more detail later in this section.

Table 10.1. Basic SqlCommand Properties

Property

Default

Description

CommandText

(None)

Contains required T-SQL query, the table name, or the name of a stored procedure to execute

CommandType

Text

Enumeration: StoredProcedure or Text (TableDirect is not supported)

CommandTimeout

15

Number of seconds to wait before abandoning the query

Connection

(None)

Addresses the required SqlConnection or DbConnection object that points to the SQL Server or other data source used to execute the query

Notification

(NULL)

Specifies the SqlNotificationRequest object bound to this SqlCommand (see discussion)

NotificationAutoEnlist

False

Determines whether the SqlCommand object automatically detects "environment" dependency objects and binds to them

Parameters

(None)

The collection of SqlParameter objects used to manage the query Parameters

Transaction

(None)

Specifies the SqlTransaction or DbTransaction associated with this SqlCommand (see discussion)

UpdatedRowSource

False

Used to keep the child row PKs synchronized post-update (see discussion)

Coding the CommandType and CommandText Properties

Visual Studio has a couple of wizards (and at least one that Microsoft does not want you to use) to help you instantiate and populate a SqlCommand instance, and you can use the CommandBuilder to create the CommandText for your action commands. However, you're on your own when it comes to building the SQL for the SELECT commands used to return a rowset. I devoted Chapter 7, "Managing Data Tools and Data Binding," to the wizard approach, as it has a number of issues that you need to be aware of before you venture too far down that road. Suffice it to say that most professional developers soon discover that they have to create their own queries with very little help from the wizards once their designs grow beyond the most elementary query/update schemes.

ADO.NET is instructed how to execute the T-SQL or named stored procedure provided in the CommandText by setting the CommandType property. This defaults to "Text", the setting required to execute a T-SQL ad hoc query. If you don't set the CommandType when executing a named stored procedure, ADO.NET throws a syntax exception when the SqlCommand is executed.

The CommandText property typically contains the T-SQL you wish to execute, as I describe later in this chapter. In SQL Server, the T-SQL you provide can also be a batch of several statements concatenated together (with or without a semicolon) that are executed in series. Of course, the T-SQL you provide must be syntactically correct and use appropriate (named) parameter placeholders.

IMHO

No, the SqlClient provider does not support the TableDirect enumerator, and since I don't recommend that approach for SQL Server, it's no big loss.

Remember, if you name a stored procedure in the CommandText, you need to set the CommandType to StoredProcedureif you don't, you'll get a syntax error when the SqlCommand is executed. I don't know how many times this exception has irritated meespecially late at night when I'm not running on all cylinders. When calling stored procedures, you'll need to append a Parameter object to the Parameters collection for each (required) input and all OUTPUT parameters. Each parameter of a stored procedure that does not have a default value assigned in the stored procedure definition must be supplied a value. You'll also need another Parameter for the RETURN value if this is of interest to your code. I discuss executing stored procedures later in this chapter.

When you set the CommandType to "Text", ADO.NET calls the system stored procedure sp_executesql to execute the SQL in the CommandText on the serverunless you set the Prepare property to True when ADO.NET executes the extended stored procedure sp_prepexec (that combines a "preparation" step with an execute step sp_execute). I discuss the Prepare property later in this chapter. The only reason I mention this is the issues caused by use of these procedures to execute your T-SQL. It seems that there are a number of perfectly reasonable (albeit more complex) queries that cannot be executed using the "Text" CommandTypethey must be executed via stored procedures to work. I'm still lobbying for a fix, but again, don't hold your breathMicrosoft is busy elsewhere working on shiny new toys.

The alternative CommandType is "stored procedure". In this case, ADO.NET simply builds a string that has the stored procedure name (the CommandText) concatenated with the input Parameter objects (by name) and executes it directly on the server.

Maximizing CommandText Performance

There are several approaches to populating the CommandText property that you need to fully understand before moving on. A bit later in this chapter, I examine ad hoc queries where you simply (perhaps not so simply) hard-code the SQL, parameter-driven queries and show how to use SqlCommand objects to call stored procedures. At this point, I want to hammer in a few guidelines to make sure your queries are as efficient as they can be:

  • Ensure that you're fetching "just enough" data in your queries and no more. Don't ask for columns you don't need or use.

  • Avoid use of SELECT*, as this syntax returns all columns currently defined for the tablepossibly more data than you need. In some cases, limiting the columns you return can result in an "index seek", which does not require secondary table access.

  • Don't ask for more rows than you need in the immediate future. Except in connectionless architectures (PDAs and phones), use the available connection to fetch more databut only as you need it.

  • Avoid queries that have to be executed with row scans or are so broad the indexes can't help retrieve the rows.

  • Don't let the user dictate how many rows to fetchat least, keep the number within reason. One greedy user should not cripple the performance of others. Use paging techniques to fetch the "nth" block of data from the server.

  • Use parameter-based queries and stored procedures whenever possible. Avoid (like the plague) ad hoc queries that accept parameters generated by the user. Remember, all data is evil until proven innocentespecially if it comes from people.

    IMHO

    All data is guilty until proven innocent.

  • Always use a WHERE clause or TOP clause (or both) to limit the number of rows returned. This dramatically improves scalability and performance, as it reduces network load and server load, and leverages existing indexes.

  • Use the query analysis tools to determine whether your query is making the best use of existing indexes. If you're using stored procedures, make sure their query plans match the input parameters each time they are called. Earlier chapters discussed these issues in depth.

  • Minimize round-trips to the server. Each time you query the server, the expense of reopening the connection, opening a channel, and all of the other overhead involved in executing the query and returning the resultsets is fairly high and should be minimized. If you can, use multiple resultset queries that return more than one set of results in a single round-trip. Think of a round-trip to the server as a trip to the grocery store on the other side of town. Make the best use of your travel time and the items you bring back. Remember that data spoils faster than milk on a warm day.

  • Avoid use of the CommandBuilder to build the CommandText at runtime. I discussed this issue in depth in Chapter 8, "Getting Started with ADO.NET."

Setting the Connection Property

Each SqlCommand object that you expect to execute must be associated with a SqlConnection object capable of executing the T-SQL in the CommandText. This property can be set using one of the New constructors or by simply providing the name of the SqlConnection object (before the SqlCommand is executed). The SqlConnection need not be open as you build the SqlCommand object, as ADO.NET won't automatically use an open connection to "guess" at the other properties or populate the Parameters collection (as ADOc did). The SqlCommand class can also accept a System.Data.Common.DbConnection object. This means you can create an alternative way to set up a valid SqlConnection object for use with your commands.

The SqlConnection object associated with the SqlCommand object is not only used to provide an access path to the data source; it's also used to generate transaction scope and Transaction objects. The SqlConnection object is also the host for the InfoMessage event, which fires when a stored procedure executes a RAISERROR or, in some cases, when a SqlException is raised. If you specify Integrated Security, the CommandText is executed on the server with the rights granted to the current user.

Setting the CommandTimeout Property

When you execute the SqlCommand object, ADO.NET passes the CommandText to SQL Server (or the designated data source) and begins to wait for the first packet of "results" to return from the server. Unless you reset the CommandTimeout value, ADO.NET waits (the default) 30 seconds for the SqlCommand to begin to return rows. No, the clock does not start counting until SQL Server responds. If the NIC or Net is down, you might wait a lot longer. In addition, if SQL Server returns a single resultset from the query and the server hangs, ADO.NET considers the timeout to be satisfied and continues to wait (indefinitely) for the remaining rows. If you need to reset the CommandTimeout value to its default setting, you can use the ResetCommandTimeout method.

If you have a query that takes longer than 30 seconds to execute, I recommend that you:

  • Reconsider your query strategy. If this is a fetch query that takes longer than 15 seconds, it's likely (but not certain) that it's returning too many rows or the procedure is not granular enough. This is especially important in interactive applications. Users won't wait forever for a query to complete.

  • If your query has to wait for another query to complete before it can complete, you've created a deadlock condition that the server might (just might) catch. In this case, you need to decide how to proceedeither execute your queries in a different order or back off and try again later.

  • Consider that you might get a "Connection timeout" exception that's returned by the connection-pooling mechanism. The Command timeout property has no effect on this behavior.

IMHO

It's not how fast you ask questions; it's how fast smart questions can be answered.

As shown in Figure 10.2, my (quite informal) studies have shown that the propensity of a person to press Ctrl+Alt+Del, smash the monitor, or take other (all-too-often) drastic actions as their wait time approaches 60 seconds really depends on the culture and the region. Some users seem to be quite happy waiting for 1545 seconds without worrying if their program will ever return. Generally, those are what Peter calls "civil serpents". However, as the wait time nears 60 seconds, most cultures (except for some monks in Nepal who are particularly laid-back) give up waiting and start taking action. These actions range in intensity from simply clucking softly, to resetting the system, to using a 12-guage shotgun on the monitor. It's interesting how users take their frustrations out on the monitor, when it was the system itself or the programmer down the hall that was more likely at fault. The chart shown in Figure 10.2 shows how this tendency varies as to locale.

Figure 10.2. Percentage of users resetting the system as wait time approaches 60 seconds.

One way to prevent users from this frustration is to entertain them. This is how Microsoft Windows handles the fact that it takes so long to simply move data from place to place. For example, when you expect the operation to take a long time, show the user a progress bar or play an interesting AVI file on pig farming downloaded from the Discovery Channel. This approach usually distracts the user enough to give your application time to get the operation finished.

Unfortunately, none of us really knows how long a query is going to execute, so some guesswork is often involved. I'll talk about the GuessQueryRuntime class in my next EBook. But seriously, you probably know how long it should take to run your query based on past experience, testing, or just an educated guess. I suggest you set the CommandTimeout property to reflect that value plus a factor to account for the stuff that happens that you didn't expect, and set up an exception handler to trap the timeout exception.

Setting the Parameters Property

Most T-SQL queries you execute include one or more input, OUTPUT, in-out, or RETURN value parameters. I call these the "gazintas" and "gazoutas", as they pass information into the T-SQL and get values back out of the resultset. The Parameters property contains the collection used to hold parameter descriptions (saved as Parameter objects), so they can be inserted into the SQL just before execution. I show you how to populate and manage the Parameters collection later in this chapter. This same Parameters collection can also collect data back from an action command (one that executes an INSERT or UPDATE, in this case). If properly configured (and I'll show you how in Chapter 12, "Managing Updates"), you can fetch the newly created row values, including the new Identity value, using these Parameter objects.

Setting the Notification and NotificationAutoEnlist Properties

One of the new 2.0 Framework features is the ability to monitor the rowset accessed by the CommandText query. When the data changes, your application can be programmed to trap a SQL Server 2005generated notification event. The Notification and NotificationAutoEnlist properties are used to set up and manage this feature. I discuss notifications along with these properties in an upcoming EBook.

Setting the Transaction Property

When you need to ensure that the operations in two or more commands are executed together (or not at all), you must "wrap" the operations in a transaction of some kind. For example, when you want to delete a customer record, you need to do so in stepsdeleting the "child" orders first and then the "parent" customer records. These operations should be done with an "atomic" operation that either completes in its entirety or does nothing at all. Generally, I suggest that transactions be carried out on the server in stored procedures. This simplifies application design and precludes interrupting the transaction (which blocks server resources) accidentally with user-interface code. I discuss transaction management in Chapter 12.

Setting the UpdatedRowSource Property

When working with parent/child relationships, one of the problems you have to face is keeping the child rows synchronized with newly added parent rows. The UpdatedRowSource property (when set to True) automatically propagates newly assigned parent PKs to related child rows once the parent rows are added to the database. I discuss parent/child relationships and managing updates in Chapter 12.

Understanding the SqlCommand Methods

Clearly, the SqlCommand class is a pivotal piece of the ADO.NET data access paradigm. As such, it's not surprising that it has so many methods (used to manage object instantiation and tear-down) and functions (used to configure and carry out the SqlCommand object's functionality). At this point, I'm ready to make you aware of these methods and properties (and not much more)I'll get into a lot more detail later in this chapter. Table 10.2 lists the methods with a brief description. All are detailed in the subsequent sections.

Table 10.2. The SqlCommand Methods

Method

Description

Cancel

Attempts to stop execution of a query. Stops further attempts to fetch the query rowset.

Dispose

Releases SqlCommand resourcesmarks the object for garbage collection.

New

Constructs a new object of the SqlCommand class. Sets the CommandText and Connection properties and Transaction.

Prepare

Used to create a "prepared" version of the SqlCommand and validate the Parameter values.

ResetCommandTimeout

Resets the CommandTimeout to its default value (30 seconds).

Using the Cancel Method

The Cancel method is used to ask (request, petition) ADO.NET and SQL Server to stop processing the operation(s) requested by the most recent Execute method (like ExecuteReader). You won't get an opportunity to call the Cancel method if you use the Fill or Update methodsthey're strictly synchronous.

If you simply close a DataReader after having used ExecuteReader to create it, ADO.NET loops through any rowset(s) generatedreturning any and all rows to the client before actually closing the DataReader (and possibly closing the SqlConnection). On the server end of the wire, your SQL Server instance is still required to finish looking for rows requested by your query.

If, on the other hand, you use the SqlCommand.Cancel method, the SqlClient data provider is requested to stop executing the current batch and sending rows back to your application. This can save a dramatic amount of time if you change your mind about the command currently running.

No, you can't expect to use the Cancel method while the query is runningunless you run the execute function on a separate thread. And then, what's the point of setting up the custom code to run an execute method on another thread? ADO.NET 2.0 has implemented asynchronous operations. I'll discuss these asynchronous functions in Chapter 11, "Executing SqlCommand Objects."

Based on conversations I've had with Microsoft, I understand that you should never call the Cancel method unless you're trying to execute a single-resultset SELECT command. Trying to stop more complex queries is problematic, at bestsomething akin to stopping an airplane's takeoff at an arbitrary point in timeonce the ground-rollout has progressed past the point of no return, it's unwise to abort. Based on how SQL Server executes queries and action commands, the Cancel method might not be received in time to stop an UPDATE or complex stored procedure. When dealing with complex queries, it's best to let ADO.NET unravel the operations. If you have started transactions to protect the referential or data integrity of your database, these should continue to do their job if you simply close the SqlDataReader.

Any number of times, I've found that SQL Server seems to lock up for long periods of time when I use the Cancel method unwisely. To make matters worse, I have been known to power-cycle the server to try to recover the system. Ah, this is not wise. Either avoid Cancel or learn to be patient.

Using the Dispose Method

I can't think of many reasons to use the Dispose method on the SqlCommand object, since object disposal is done automatically. However, if you create a SqlCommand object and want to ensure that its resources are released to the garbage collector, you can use the Dispose method. When you learn how to use the Using statement to help declare your SqlCommand (or any) objects, you'll find that it calls Dispose for you as the object falls from scope. I'll show examples of the Using statement later in this chapter.

Using the New Constructors

There are four New constructors implemented on the SqlCommand object. Ah, no, they aren't really "new", so if you're confused by that term, you must be new to OO (so to speak), so it would be a good idea to review Chapter 8, where I discussed New instance constructors. These are used to instantiate an instance of the SqlCommand class and (in most cases) populate the CommandText and, optionally, the Connection and Transaction properties. No, when you supply a string to the constructors that expects appropriate CommandText SQL, table, or stored procedure names, the constructor does not edit the stringit assumes you know what you're doing. Syntax checks are made by SQL Server when the CommandText is executed. ADO.NET also does not automatically guess that you're actually passing the name of a database table or stored procedurethat means you'll have to set the CommandType property yourselfit's a common mistake, so be sure to check before executing. Figure 10.3 shows typical uses of the SqlCommand New constructors.

Figure 10.3. Using the New constructors to build SqlCommand class objects.

[View full size image]

If you look closely (perhaps not that closely), you might spot several issues with this code.

  • The first constructor simply creates a SqlCommand object, but it's not ready to be executedit's up to your code to populate the required properties before it can be used.

  • The second constructor attempts to set up a parameter query but fails to populate the Parameters collection with a properly defined Parameter object to manage the query parameter. You also get extra credit if you noticed that the "?" is an invalid parameter placeholder when using the SqlClient provider. All parameters (as I'll discuss later) must be namednot simply marked as in ADOc, OleDb, or Odbc, where one marks query parameters with "?".

  • The third constructor attempts to call a stored procedure "AuthorsByYearBorn" and pass in the current SqlConnection. So far, so good, but when calling a stored procedure, remember that you must set the CommandType property to "StoredProcedure" so ADO.NET knows how to execute the CommandText.

  • The fourth constructor example is syntactically correctit illustrates passing a string to the CommandText string in the New constructor. However, it also uses a silly way to determine how many authors were born before the year 1000. Wouldn't a SELECT be far easier and not require a transaction? Sure, but I wanted to illustrate use of a transaction.

Let's take another attempt at the code and see if it can be improved a bit. In this next version, I simply dropped the first constructor, as it did not do anything usefulit was kinda like a snowplow on a skateboard. Figure 10.4 shows the results of my efforts.

Figure 10.4. Using the SqlCommand New constructors.

[View full size image]

Tip

Parameter-driven queries are your best defense against SQL injection attacks.

Note that in Figure 10.4, I used a correct parameter marker for the ad hoc query (@AuthorID). I also used parameter markers that matched the named parameters being passed to the stored procedure. I'll get into the details of how to build the SqlParameterCollection later in this chapter. I also ripped out all of that (needless) transaction code and executed a simple SELECT to return a count of all rows that qualify for the WHERE clause. To return the result, I used ExecuteScalar to return the first column of the first row returned by the query as an object. I'll talk about the SqlCommand "execute" methods in Chapter 11.

Using the Prepare Method

The Prepare method was first introduced in the ODBC days when it was necessary to "prepare" SQL statements for execution. The Microsoft documentation (and the MCSE exam) seems to think that it's a good idea to call the Prepare method the first time a SqlCommand is executedI think so, too. One thing that's certain is that if you call Prepare, all variable-length parameters must have a non-zero size set and have explicit datatype definitions. This means you can't use the new ADO.NET 2.0 AddWithValue function to add a Parameter to the Parameters collection (it assumes the default datatype of nvarchar).

As I mentioned earlier, the Prepare method calls the Extended Stored Procedure (ESP) sp_prepexec, which combines the sp_prepare and sp_execute ESPs. These procedures create a cached version of the query plan (to ensure that it's reused when the SqlCommand is executed again) and execute the SQL. Using Prepare can mean a performance benefit once you pay the price of calling itPrepare is not free. While Prepare does not require an extra round-trip, it does incur some additional timewhich does not have to be spent on subsequent executionsassuming you make subsequent calls using the same SqlCommand.

Using the ResetCommandTimeout Method

When you need to set the CommandTimeout back to the default value (30 seconds), you can use the ResetCommandTimeout method or simply set the CommandTimeout property to 30that's what ADO.NET does behind the scenes.

Understanding the SqlCommand Support Functions

This discussion focuses on two "support" SqlCommand functions, as listed in Table 10.3. Since the Clone function is new in the 2.0 Framework and has so many useful purposes, I'll spend quite a bit of time helping you make best use of its features.

Table 10.3. The SqlCommand Synchronous FunctionsPart 1

Function

Returns

Description

Clone

SqlCommand

(2.0) Returns a new instance of the SqlCommand object in its current state, including all property settings and Parameters collection

CreateParameter

SqlParameter

Constructs a new instance of the SqlParameter class (no constructors)

Using the Clone Function

The Clone function is fairly simplelike most Clone functions in .NET, it's used to make an exact duplicate of the object instance. Figure 10.5 illustrates how the Clone function can be used in code. In cases where you want to create several different Command object instances, you can use the Clone method to copy the populated "base" instance to create a new instance. In some of the examples shown later, I use the Clone method to make a copy of a DataRow or an entire DataTable. The magic of the Clone method is that it makes a copy of the datanot just set a pointer. This means you can have two independent sets of objects to work with.

Figure 10.5. Using the Clone function to replicate a SqlCommand object.

[View full size image]

Take a look at the code in Figure 10.5 [1]. Do you see any issues? As far as this goes, I expect that it might make more sense to pass in the @YearWanted parameter as an argument to the BuildCommandAndClone function. But that raises another point. Should you instantiate the SqlCommand object each time it's used? Object instantiation is not expensive, but it's not free, either. Let's take another look at how the Clone function can be used a bit more efficiently. In this case, I'm going to use an "overloaded" subroutine that has two "signatures"one that accepts a single integer parameter and another that accepts two integer parameters. I'll build a SqlCommand object, clone it, and add the additional parameter to the clone.

[1] See the "Clone Function" example on the DVD.

First, I set up the objects I plan to use in the application. In this case, I create a SqlConnection object and two SqlCommand objects, as shown in Figure 10.6. Note that I extend the existing CommandText to incorporate another argument in the WHERE clause. Yes, I could use the Parameters.Clear function to clean out the Parameters collection, but this way I have to add only one additional parameter instead of two. Note that the "@YearWanted" parameter is used twice in the SELECT statement's WHERE clause in the cloned SqlCommand object. You're permitted to use parameters as many times as necessary or not at all in your T-SQL.

Figure 10.6. Cloning a SqlCommand object and creating a SqlConnection.

[View full size image]

In Figure 10.7, I create two subroutines that are themselves "clones" of each other (in a way). These subs are "overloaded," which permits the developer to create more than one way to call the routinewith one or two arguments. In one case, I pass a single integerthe "year wanted", and in the second case, I pass twothe "year low" and "year high" to be passed to the "Between" operator in the SELECT statement's WHERE clause. The GetAndShowData routine does just that. It accepts one of the two SqlCommand objects (with the Parameter Value properties already set) and executes the query. The results are bound to a DataGridView control.

Figure 10.7. Create two overloaded functions to call the right SqlCommand and display the data.

[View full size image]

Before I move on, consider that it often makes sense to create your SqlCommand objects early in the application and reuse them for the lifetime of the application. In other words, you create them once and simply set the Parameter.Value property just before each execution.

Using the CreateParameter Function

One approach to creating SqlParameter objects is to create them individually and populate their properties one by one. This is (IMHO) a waste of time, as it's far easier to simply use the numerous overloads exposed on the SqlParameterCollection class that I discuss later in this chapter.


Integrating Ad Hoc Queries into Your Application

An ad hoc query is simply a T-SQL statement or a batch of several statements that you hard-code into your application. Generally, ad hoc queries are provided as a properly formatted SQL string. No, this does not mean you need to add extra line-feed or carriage-return characters to the string to make it look nice. Sure, this T-SQL string can be built separately and used to set the CommandText property or used with the SqlCommand New Constructors.

The CommandText string can be built on-the-fly, and you can concatenate the query components one at a time, as shown in Figure 10.8but before you start pounding in this code[2] or loading up the DVD, let's take a closer look at the process of creating and managing ad hoc queriesand the litany of problems they introduce. The point of this exercise is to ensure that you understand the issues and learn to avoid this approachdespite the fact you'll see it illustrated throughout this book and most other books and documentation. I tend to use this approach as it's the best way to illustrate integration of T-SQL queries into your application. In most cases, developers move from embedded T-SQL in their code to stored procedures once they get the database schema and procedures created. The T-SQL embedded in your application will be a good starting point for these procedures.

[2] This sample is named "SqlCommandTextExamples" on the DVD.

Figure 10.8. Creating a SqlCommand.CommandText with concatenation.

[View full size image]

As I complete the final draft of my EBook, "Hitchhiker's Guide to SQL Server Everywhere", I find the use of concatenated queries even more important than ever before. Consider that SSEv does not support stored procedures so all of the queries you executeespecially parameter-driven queries, are going to involve concatenation on on-the-fly SQL construction of one kind or another. I've been asked any number of times if using ad-hoc queries hurts performance. Frankly, it used to, but given the way that SQL Server caches its query plans, you'll find that there is very little performance benefit from a properly constructed parameter query. The key phrase here is "properly constructed". If the query parameters substantively change how the query would be created, subsequent invocations of a query can perform poorly because the cached query plan does not efficiently return rows given a different set of parameter values. I talked about this in Chapter 2, "How Does SQL Server Work?," so you might want to revisit the section that discusses how the query optimizer works.

As you decide which coding strategy to use when building concatenated strings to pass to the CommandText, consider the following alternatives that balance ease of use, human readability, and performance:

  • Concatenate using the language-specific "&" (or "+" in C#). This approach is not always as fast as using the StringBuilder Append techniques but can be just as fast for small stringsand easier to code.

  • Concatenate using the String.Format method. This provides some datatype conversion but is limited in depthyou can only concatenate a few arguments.

  • Concatenate using the StringBuilder Append or (better yet) AppendFormat methods. These can improve performance if the strings are fairly long.

The following example illustrates the first suggested concatenation technique and suggests alternatives. Remember that the performance gained here must be balanced with the ability to clearly see what's being sent to the CommandText to help others maintain the code you write.

Yes, there are several problems with this code. Can you spot them all? No, I'm not talking about the fonts. Let's step through the issues:

  • First, the "&" concatenation operator is somewhat expensiveit's slower and trouble-prone when compared to other techniques. While it's fine for short strings, each time it's used, the string is copied several times behind the scenes. While book authors sometimes have to illustrate long queries by breaking them up like this, it's not a good idea at all to write your programs like this. Remember, it's the computer that has to read these queries most oftenmaking them easier for you to read is also important, but balanced with the expense of code execution performance. If you insist on using ad hoc queries, I suggest using the StringBuilder class or String.Format to construct them. I'll show you how in a minute.

  • When you concatenate values into the WHERE clause, as shown in Figure 10.8, you open yourself (as in, you, personally) for attackthat is, when your company data is penetrated or destroyed because of a SQL injection attack, everyone is going to blame you. If you're lucky, you'll just lose your job. If you work for some of the folks I know in Texas, you might not make it out of the parking lot with your dignity intact.

  • In this T-SQL, the LIKE expression requires a quoted string. Do you see any single quotes to frame the inbound strCustWanted string? The application assumes (quite foolishly) that the user will somehow know to enter the single quotes around the customer name they want.

  • Okay, let's say the user is (by some miracle) smart enough to frame the customer's name in single quotes. But what if the customer's name is "O'Malley"? Try it and see what happensyou'll get a syntax exception, and so will they. Okay, in this case, you need to train the user to double-up any single quotes embedded in a name they're searching for. Yes, you can scan the string and use the Replace method to substitute each single quote ("'") with two single quotes ("''"). Another approach is to attempt to remove all of the customers in the database whose names have embedded quotes. This was tried in a northern European countryit didn't work very well.

    Tip

    I call the embedded single-quote problem "The O'Malley Issue."

  • And what about the LIKE expression wildcard character? If you don't use one, the LIKE expression returns no rows. Right, in this case, you'll need to train the user to add a "%" to the name in the correct placeno, not "*", but "%". As it is, your training manual is getting kinda thick and the users are going to be a bit grumpy when told they need to learn all of these rules. So, you can (and should) add the wildcard character yourself, if it's not already there or is in the wrong place.

Let's go over several additional points to help you avoid trouble if you still want to use this ad hoc query approach.

  • Did you notice each line of the T-SQL is concatenated with an extra space added at the beginning? I began teaching this technique years ago when my student's concatenated queries failed with syntax errors. It seems that the T-SQL parser did not understand "CustomersWHERE". Remember that extra whitespace is ignored by T-SQL.

  • Consider that the flexibility of this approach is one of its strengths. You can build up a T-SQL statement that can specify the columns, tables, and entire WHERE-clause, as well as ORDER BY expressions. T-SQL will let you supply only WHERE clause constraints in a parameter query.

  • When you hard-code T-SQL in an application, if that query has to change even in simple ways, you're looking at a recode, retest, and redeploy, as well as possibly retraining your staff every time it needs to change. Wouldn't it be nice if you could make simple (or even moderately complex) changes to the queries without recoding? I'm about to talk about that optionit's called using stored procedures.

  • Notice that the T-SQL asked for specific columnsI didn't use SELECT *. As I've said before (and I'll say again), I think SELECT * has its place, but not in most queries. If you're dumping a table schema, sure, use it, but if you're building a query, return only the columns you need and no more. This helps eliminate performance bottlenecks when SQL Server is forced to return columns you (or your user) don't need or don't expect.

  • In this example, I opened the pre-instantiated SqlConnection object just before I executed the query. I also made sure the SqlConnection was closedeven if something went wrong (as it often does)by putting the Cn.Close in the Finally block. This code gets executed whether or not the code between Try and the first Catch works. It's a common mistake to place the Cn.Close just below the code that sucks the rows out of the DataReader. As I'll show you later, you shouldn't close the SqlConnection associated with a DataReader until you're done with the rowset.

  • Did you notice that I included an exception handler to trap the SqlException (that I really expect to get) in addition to the general Exception that I don't expect to get? And, no, I don't recommend showing any of these messages to the userI don't expect the average off-the-street user is capable of figuring out what to do to solve the problem (unless that street is NE 156th in Redmond in front of Building 40 on the Microsoft campus).

  • Last, but not least, consider that the O'Malley issue is only one of the many hidden issues you need to be aware of if you choose to roll your own ad hoc queries. You'll also have to deal with (complex) date formatting and framing, datatype assignments, and inbound data validity checks, among others. But far and above SQL injection is the most serious issue you need to be constantly vigilant to prevent. As a rule of thumb, you should never accept unvalidated strings from users if these are to be concatenated into your SQL. However, even complex validation routines have been fooled. It's far (far) safer to use parameter-based queries, as they can foil most SQL injection attacks.

Tip

How many of these issues did you catch? Once you can spot most of them in the code you're reviewing or writing, you're ready for the big leagues.

Preventing SQL Injection Attacks

I'm not going to give you (or anyone) the sequence of characters used to inject destructive (or simply intrusive) SQL into your queries. Suffice it to say that SQL injection is a very widely practiced and, sadly, widely published and discussed technique that permits hackers to gain full access to your server and do what they will. Consider that SQL injection attacks usually focus on the ability of SQL Server (and other more sophisticated DBMS SQL engines) to execute more than one statement in a single batch. That is, a SQL query can contain several (N) operations. While your query might only intend to return a few rows, SQL injection attacks "inject" additional (malicious) SQL into your benign query. Incidentally, because SSEv does not support multiple statements in a SQL query, it is not as vulnerable to SQL injection attacks.

There are several ways to protect your database, your company, and your career. All of these are fairly easy to implement, but let's discuss some broader security guidelines that reduce the frontal area of your server's security risk.

IMHO

Sadly, there seems to be no end to the number of articles that clearly show how to perform SQL injection attackssome by companies simply trying to sell injection-prevention software. It's kinda like lawyers drumming up business by dropping banana peels on the sidewalk.

  • Use the SqlCommand class to manage the query and any parameters it requires. Because ADO.NET knows how to deal with string and date framing and formatting issues, your code can actually be simpler to write. As another side-effect of the way ADO.NET handles parameters and executes the query, you and hackers will find that it's virtually impossible to insert anything into your SQL via the parameters.

  • Treat all input and all new data as "evil" and untrusted. Okay, this sounds a bit paranoid and rings of a Homeland Security tactic, but that's where the industry has evolved. As a rule of thumb, I never permit unvalidated data from entering the system in the first place. Knowing where the data comes from is not enough. Hacking can result from human input or spurious data introduced by a trusted programthat perhaps has been hacked. Remember that over 80% of all security breaches are from inside your firewallfrom your coworkers and those interlopers that have gained access to idle systems inside your buildings.

  • Avoid ad hoc or system-generated "dynamic" SQL either in your application or in a stored procedure. Concatenated SQL is the first route a SQL injection attacker will attemptit's the ground-level bedroom window left open to catch the summer breeze. These hacking attempts can be made as you capture a Login ID or the "Product Wanted" search argument.

  • Practice "least privilege" security: That is, set up specific accounts for your applications and users that grant only the specific rights they need. Given the credentials you're using (those provided by the Windows login account or IIS, or those hard-coded in the application), hackers running injected SQL with those rights can't do any damage or extract protected data. Don't put all of your data eggs in one basket. By this, I mean don't create a single application login that has broad rightsremember to provide just enough rights to get the job done.

  • Use encryption for all sensitive data: Make sure that credit card numbers, Social Security numbers (or the equivalent), and other private data is encrypted in place. Enable the SSL option in SQL Server to prevent hacking data feeds. Of course, enabling SSL won't help if the server is not configured to support encryption.

  • Don't depend on filtering algorithms that attempt to ferret out those keywords known to trigger SQL injection. While this might work in some cases, they're vulnerable to the seemingly infinite patience of hackers to figure out ways to bypass the filters.

  • Don't reveal the secrets of the inner sanctum by dumping your exception messages to the user. Remember, the user is the person least able to fix your problem, but a hacker is the person most likely to compromise your system with the inside information revealed by an Exception.ToString dump.

In your ASP.NET application, when an unhandled exception is generated, make sure only minimal help is offered to the hacker. This is accomplished by setting the debug attribute of the compilation element (in the Web.config file) to False and setting the mode attribute of the customErrors element to either On or RemoteOnly. For example:

<compilation defaultLanguage="c#"
 debug="false"
<customErrors mode="RemoteOnly" />

The RemoteOnly setting will ensure that users accessing the site from localhost will get informative error messages, while those accessing the site from a remote location will receive generic error messages that reveal no useful information about the exception. Use the On setting to have all users, including local users, see the generic error messages. Never use the Off setting in a production environment.

Building Safer SqlCommand Objects

When you build your SqlCommand objects, you can prevent SQL injection attacks (or at least reduce them) by

  • Using strongly typed data fields: Don't use string datatypes for any user input that can be expressed as an integer or other number. This means you're going to have to take the TextBox, ComboBox, or other control's Text property and cast it to the correct type.

  • Using SqlParameter objects (which are strongly typed) to manage all values passed to the server. In other words, don't pass any string to the server as a concatenated value in a SQL statement. Because of the way that ADO.NET executes parameter queries, it's virtually impossible to pass stray SQL by way of a Parameter.

  • Calling stored procedures that further filter the data based on business rules. Stored procedures can also limit the length of the inbound string parameters to help reduce the chance that extra values are introduced on obscure parameters.


Coding Parameter Queries

Okay, so do you want to have many of these ad hoc query issues magically disappear? If you're considering use of parameter queries, you've come to the right place. The ADO.NET parameter query is designed to help address several complex issues you're already aware of and several issues you might not even realize existuntil they bite you in the butt just as you sit down for dinner some evening. T-SQL and ADO.NET are tuned to help developers build parameter-based queries, and Visual Studio can help. No, this does not mean you have to graduate to stored procedures right away. I sometimes prototype applications with parameter-based ad hoc queries even though I know I'm going to convert to stored procedures some time in the future. Let's step through the process of creating a parameter-based ad hoc query. From there, we can easily step into calling stored procedures.

A parameter-based query is not that mysteriousit simply passes the job of inserting the properly formatted and framed string into the T-SQL just before it's sent to the server. In this case, ADO.NET assumes the responsibility of framing string and date query parameters in single quotes, if necessary, formatting any date values, and dealing with Unicode and internationalization/localization issuesjust to name a few.

While ADO.NET's Parameter object handlers know how to deal with the O'Malley issue and can help eliminate SQL injection attacks, they can't deal with all of the issues I described earlier. For example, ADO.NET can't automatically insert the "%" in the correct place in your LIKE expressionyou'll still have to do that.

As with any query you build, you're going to want to test it (well, I hope you will) before you cast it in bronze. While Visual Studio has a number of useful tools, it supports only a subset of the tools you'll find in SQL Management Studio. There, you'll find a Visual Studiolike interface (well, sorta like Visual Studio), a full-featured object browser that exposes all of the objects, rights, and utilities needed to manage your database, along with a query editor/analyzer to help debug the T-SQL. The problem with this editor/T-SQL debugger is that it sucks (that's a technical term). Once you learn how easy it is to write and debug code in Visual Studio, it's tough to go back to a stone-age tool like Query Analyzer.

Once you've coded and tested the query, you're ready to commit it to your application by either hard-coding the T-SQL into the SqlCommand.CommandText or taking the next big step and creating a stored procedure to manage the query. I don't think we're ready to take that leap just yet.

Inserting Ad Hoc Queries

Your next little challenge comes when you find that you must incorporate hand-written T-SQL into your source code. Microsoft has still not provided a tool to do this easily. Their own attempts to incorporate generated SQL into their behind-the-scenes classes are pretty sad. Sure, it works, but the code is far from readable, which makes it hard to understand and maintain. When you first paste your working SQL into the Visual Studio code editor, it looks something like Figure 10.9. Notice that Visual Studio has already started to syntax check the inserted text.

Figure 10.9. Pasting T-SQL into your Visual Studio code editor.

At this point, you must decide if you want to use the "&" operator to connect these loose strings into a single string, use the StringBuilder class, or simply use the String.Format method. Unless you're concatenating a few small strings, the StringBuilder performs better. Figure 10.10 shows how to use the StringBuilder instance to build up your T-SQL query string. Note that it's not necessary (or advisable) to break up the T-SQL string into many parts like this. While I have to do so to fit the code on the printed page, you should not worry about human readability that much.

Figure 10.10. Using the StringBuilder to construct an ad hoc query.

[View full size image]

You'll want to add an Imports statement to your Visual Basic application to add the System.Text namespace unless you're being paid by the keystroke. As you use the StringBuilder Append method, be sure to add whitespace between the T-SQL operators so the string will execute. No, you do not need to add carriage-return and line-feed characters to the stringthis makes a difference only if you're coding a stored procedure. While this example is pretty simple, you get the idea. Sure, it could have been done with "&", but many queries are far longer than this, so getting accustomed to using the StringBuilder is an important skill to learn.

Another approach that can make coding easier (and less prone to concatenation errors) is to use the String.Format method to build up the T-SQL query. This approach works fine for simple queries or anytime you need to insert a value into a string as shown in Figure 10.11. Since Peter twisted my arm about this a couple of years ago, I've grown to like the ability of String.Format to deal with a litany of formatting and other issues.

Figure 10.11. Using String.Format to construct simple ad hoc queries.

[View full size image]

Once again, regardless of the technique you use, it's dangerous to concatenate user-provided arguments into T-SQL strings, as it exposes your application to SQL injection attacks. Not only that, but if you insist on this approach, you'll have to be extremely careful about properly formatting the T-SQL, including dealing with framing dates and strings as well as the O'Malley issue. Nowadays, I simply can't trust users to provide "clean" parameters. Remember, all data is guilty until proven innocent.

Understanding the SqlParameterCollection Class

Unless you're executing very simple queries or action commands that don't accept user-supplied values, you should code a Parameters collection to handle the values passed to query input parameters, as well as capture the values passed from the OUTPUT and RETURN value parameters. As I've said before, the SqlCommand is capable of dealing with a litany of issuessome of the most complex involve handling parameters. The Parameters collection holds some or all of the Parameter objects that map to the parameter placeholders in your T-SQL. When you execute the SqlCommand, the Parameters collection is scanned (by name) to find Parameter objects that match the named placeholders. If a Parameter Value is set to Nothing (in Visual Basic .NET) or null (in C#), the Parameter is skipped and not sent to SQL Server. This assumes that the missing parameter(s) have defaults defined in the stored procedure. Of course, this won't work for ad hoc queriesyou'll have to provide Parameters with Values set for all query parameters.

After the query is executed and all rowsets (if any) have been fetched, any OUTPUT and RETURN value Parameter.Value properties are populated. It won't do any good to access these Parameter object Value properties until the last rowset of the last resultset has been fetchedor you use the Cancel method, as described elsewhere in this chapter.

Exploring the SqlParameterCollection Methods

The SqlParameterCollection class has all of the typical Collection methods and functions that can be used to manage the collection itself. Those that are most interesting (and most useful) are listed in Table 10.4. Note that the SqlParameter Collection is based on the System.Collections.Arraylist and supports all of the inherited methods and functions used to manage the array.

Table 10.4. Selected SqlParameterCollection Methods

Methods

Accepts

Description

AddRange

Array or Parameter array

(New for 2.0) Permits you to create a System.Array or Parameters array and add a block of Parameters in one operation

Clear

(no arguments)

Removes all SqlParameter objects from the collection

Using the AddRange Method

I don't really see a lot of utility here, but it comes included with all other ArrayList Collection classes in the 2.0 Framework. Why create and populate a separate array of Parameter objects when you can just as easily create a Parameters collection and populate it directly? If you do find a need to use AddRange, Figure 10.12 shows how to set up an array of Parameter objects, populate it, and use the ParametersCollection AddRange method to add the pre-configured Parameter objects to the collection. Note that AddRange requires a fully populated array of Parameter objects.

Figure 10.12. Using the AddRange method to add an array of Parameter objects to the Parameters collection.

Using the Clear Method

If you want to reuse the SqlCommand object, you might consider using the Clear method to zero out the Parameters collection. Frankly, I think it's better to simply create a new SqlCommand object and repopulate the Parameters collection.

Understanding the SqlParameter Class

The SqlParameter class is (as I've said before) used to describe the named query parameters that match the parameter placeholders used in the CommandText query. Each SqlParameter you create is appended to the SqlParameterCollection. Let's look at some basic facts about the SqlParameter class:

  • Each Parameter object must be assigned a unique name that matches the placeholder name in the T-SQL. No, you can't use "?" as an SQL parameter placeholder. That technique is used with OleDb and Odbc providersnot SqlClient.

  • Named Parameter objects can be added to the Parameters collection in any order. Positional parameters are no longer supported.

  • When building a Parameters collection for an ad hoc query, all query parameters must have a corresponding Parameter object appended to the Parameters collection.

  • Before executing the SqlCommand containing an ad hoc query, each input parameter must have the Value property set.

  • When building a Parameters collection for a stored procedure, you must add a Parameter object for each parameter that does not have a default value set in the stored procedure. Yes, you can provide "override" Parameter objects that set a Value other than the stored procedure default setting.

  • If a named Parameter is missing from the collection or if the Value property is set to Nothing (Visual Basic .NET) or null (C#), the stored procedure default value setting is used.

  • When defining a Parameter object, you need to supply only the ParameterName and Value properties prior to execution. The default parameter datatype is nvarchar.

  • If you plan to use the Prepare method (for better performance), you'll have to provide a specific datatype for all Parameter objects and a specific (maximum) Size property setting for all variable-length datatypes (such as varchar, nvarchar, char, nchar, varbinary, etc.).

  • While you can define the Precision and Scale properties for a Parameter object, these values are basically ignored by ADO.NET.

  • When defining Parameter objects, the default Direction is Input. If you're calling a stored procedure that returns OUTPUT parameter(s), you'll need to create Parameter objects with the Direction property set to Output.

  • All stored procedures return an integer upon completionwhether the stored procedure sets the integer RETURN value or not. By default, this value is 0. To capture the RETURN value, you must create an (optional) Parameter object with the Direction set to ReturnValue.

  • If the Parameter can accept a null value (since all SQL Server Parameters can accept null, this should read "if you plan to pass a null to a Parameter Value), you must explicitly set the IsNullable property to true.

  • When using the Parameter object in conjunction with the SqlDataAdapter or TableAdapter, Update method, you can map the Value property to a specific column in the DataTable by setting the SourceColumn property. This is discussed in more detail in Chapter 6, "Building Data Sources, DataSets, and TableAdapters," and Chapter 12.

What Can Be Parameterized?

Ah, ADO.NET (and T-SQL) won't permit you to parameterize just anything. T-SQL insists that the column, table, and other object names be hard-coded along with most of the rest of the query. What can be parameterized are arguments passed to WHERE clause expressionsand then again, not all of these expressions can accept parameters. You can also parameterize the items in a VALUES list, as when using the INSERT statement or in a SET statement as a placeholder for the column value. Generally, parameters can be used as a placeholder for a literal value. That is, when the T-SQL is executed, the placeholder is filled with the Parameter Value property. Typically, you'll see parameters in a WHERE or HAVING clause search expression. For example, you can provide a parameter for:

  • A TOP expression

  • A LIKE expression

  • The equality and comparison expressions

  • An INSERT or UPDATE statement VALUES list

  • An assignment statement, including SET and SELECT

As I'll show you later in this chapter, you can resort to using the T-SQL EXEC operator to pass any and all of the components of your SQL query to the server to be compiled and executed. The approach is not by the furthest stretch of the imagination recommended for those who don't fully understand its performance and security vulnerabilities.

Marking Parameters

In SQL Server stored procedures, you must mark parameters with named placeholders prefixed with the "@" characterno, you can't use "?" or ":" to mark parameters, as you did with ADO.NET classic. Ad hoc queries are no different in this respect. Figure 10.13 illustrates typical parameter placeholders in an ad hoc query. This T-SQL could just as easily be the contents of the As clause in a stored procedure.

Figure 10.13. Passing parameters into a SELECT query.

[View full size image]

No, the IN expression cannot accept a parameter, due to SQL Server stored procedure compiler restrictions. One approach to get around this limitation is to pass the list of acceptable values (which is a comma-delimited list of numbers or quoted strings) to a function that returns a table-type variable that feeds a SELECT statement for the IN expression. Get it? Well, if not, Figure 10.14 shows an example of how this is done. Later in this chapter, I'll show another (less reputable) approachdynamic SQL; in Chapter 13, "Managing SQL Server CLR Executables," I'll show you how to do this using a CRL function. First, let's see how to use a table-valued function without a parameter.

Figure 10.14. Passing a delimited list to an IN expression by using a Table-valued function.

[View full size image]

Figure 10.15 illustrates how the code replaces the literal string in the function call with a parameter. I also wrote a test script to make sure it works. The script, like your ADO.NET SqlCommand, sets the SqlParameter Value property before executing the query.

Figure 10.15. Replacing a literal string with a parameter.

The table-valued function shown in Figure 10.16 is a block of custom code that I wrote to convert a delimited string to a set of rows in a data table. This table is returned by the function with one row for each item in the delimited string. The function accepts a delimited string up to 4,000 characters long and a delimiter that's used to delineate the items in the string. Once you have a table, you can use a nested SELECT with the IN expression, as shown in Figure 10.15.

Figure 10.16. The StripToTable table-valued function.

[View full size image]

Tip

Would it be more efficient to implement this T-SQL function with a CLR function? I'll discuss that possibility in Chapter 13.

Another use for the SqlCommand object's Clone function is to create instances of a SqlCommand that does not include SqlParameter objects in the Parameters collection that are supplied by a stored procedure's parameter default settings. In other words, when you call a stored procedure that has defaults set for one or more parameters, you can simply set up a clone of the SqlCommand that does not include these Parameter objects in the Parameters collectiona "sparse" parameter list.

Another approach would be to set up your SqlCommand object's Parameters collection with Parameter objects whose Value property is set to Nothing in Visual Basic (and "null" in C#). This tells ADO.NET to skip the Parameter, which forces the stored procedure to use the defined default.

Setting Default Parameter Values

Whenever you call a stored procedure, you must supply all input (or input/output) parameters specified in the stored procedure's signatureunless the stored procedure parameter has a default value assigned, as shown in Figure 10.17, where the stored procedure defines default values for the first and third parameters.

Figure 10.17. Setting default values for input stored procedure parameters.

So, to configure a Command object to call a stored procedure that includes default parameters, the Parameters collection should be configured to either

  • Build a Parameters collection that includes all stored procedure parameters and provide specific Parameter.Value settings for each parameter. At execution time, when you supply a specific Value property value, it overrides the default value defined in the stored procedure. Consider that if you set the Value property to "NULL" (DBNull.Value), NULL is passed as the Parameter value. I discuss other ways to set the Parameter Value to NULL later in this chapter.

  • Build a Parameters collection that includes all stored procedure parameters and provide specific Parameter.Value settings for parameters that should override any default settings, and set those that should take the stored procedure-defined default value to Nothing (Visual Basic .NET) or null (C#).

  • Build a Parameters collection that includes only those stored procedure parameters that do not have default values defined in the stored procedure (the required parameters) and any other parameters that you wish to provide (that override the default). Set the Value property of all of these Parameters.

  • Build one or more copies/clones of the SqlCommand that define hard-coded defaults (Value property settings) for selected stored procedure parameters and require just-in-time Value property values.

Let's walk through an example of these techniques. As shown in Figure 10.18, I set up two SqlCommand objects. The first is configured with a Parameters collection to accept a single Parameter. In this case, I assume that the application wants to use the stored proceduredefined default setting for the other two stored procedure parameters.

Figure 10.18. Setting up two SqlCommand configurations.

[View full size image]

When it comes time to execute the SqlCommand objects I've created in Figure 10.18, the application chooses between one of two routinesone that simply sets the required parameter and another that forces the first and third parameters to use the stored proceduredefined default value. Let's see how this is implemented in Figure 10.19.

Figure 10.19. Setting Parameter.Value determines whether stored procedure default is used.

[View full size image]

Building a Simple Parameters Collection

Once you get your T-SQL properly formatted, it's time to build the Parameters collection. In this case, I have two ad hoc query parameters to deal withone passed into the Sub (the delimited list of prices) and the maximum rows to return from the query, which is hard-coded as 10. But the code shown in Figure 10.20 has a few problems. Can you spot them?

Figure 10.20. Building a parameter-driven querywith some issues.

[View full size image]

The first problem you'll encounter when building concatenated SQL queries involves whitespace. Remember that when you build T-SQL strings by concatenating or appending to the StringBuilder, you need to be vigilant of the whitespace between the operators. Note that the "FROM" clause is not prefixed with a space. This means the T-SQL submitted will be "...dsValueFROM StripToTable...", which generates a T-SQL syntax error.

The next problem is that there is no exception handlinganywhere. While it's possible to set up an exception handler on the routine that calls this function, you really need to remember to set it up somewhere or the end user has to deal with unhandled exception messages.

See anything else? There is a hint in Figure 10.20. See the squiggly line under the "dt" near the end of the routine? In Visual Studio, this green squiggly is a sign that something might be wrong. In this case, it's a warning that the object being referenced has not been initialized. Remember that when you create a variable cast as a datatype, that does not create an instance of the objectit simply creates a pointer to the place in memory where it's going to be stored. You still need to create a new instance of the object or expect the method to do it for you. That's what happens with the SqlDataReaderthere is no way to create a "new" DataReader streamonly the ExecuteReader method can do that.

What about the connection? Don't you think that the SqlConnection (cn) should be closed before the end of the Sub? Well, it's not instantiated in the Sub scope, so the SqlConnection is not closed by the GC when I leave the Suband you should never depend on that behavior anyway. The SqlDataReader (dr) is closed by ADO.NET because I used the Load method to complete population (as I'll discuss later). Briefly, any method or control that completes DataReader rowset population (like any complex bound control) closes the DataReader. But that does not close the associated SqlConnection. In this case, I changed the state of the SqlConnection object but did not change it back. What happens when this routine is called again? If I don't fix it, it will fail, as the SqlConnection is already openunless some other part of the application closes it. It's always a good idea to leave object state as it was when the routine started.

Let's look at the corrected code in Figure 10.21. The green squiggly has gone, and I added the CommandBehavior.CloseConnection option to the DataReader to tell ADO.NET to close the SqlConnection when the DataReader is closed. I also added SqlConnection.Close to the Finally block to ensure that the SqlConnection gets closedwhether or not there are exceptions. Sure, it's okay to close a closed SqlConnection.

Figure 10.21. The corrected subroutine to fetch rows based on input parameters.

[View full size image]

Constructing the Parameters Collection

When I build the Parameters collection, I start from a clean slatean empty collection. To add Parameters, I can use any one of a variety of syntax variations. Since the code is using named parameters, they can be appended to the Parameters collection in any order. When I start talking about stored procedure parameter lists, there are a few more considerations to worry about, but now let's get the basics down. There are (at least) three ways to fill the Parameters collection with Parameter objects.

  • The Add function: Appends a Parameter object to the Parameters collection using one of four overloads (constructors). This method permits you to pass in a pre-built SqlParameter object or set the name, type, size, and source column of a newly constructed Parameter object.

  • The AddWithValue function: This function (new with the 2.0 Framework) appends a Parameter object to the Parameters collection by simply passing the Parameter name and value. The datatype defaults to nvarchar.

  • The AddRange method: This method (new with the 2.0 Framework) appends either a typed array of pre-built SqlParameter objects or a system.array (but it can't accept the Parameters collection of another SqlCommand).

The Add function is used by the wizards (that I discussed in Chapter 7) coupled with one of the seven Parameter class New constructors. These are a lot harder to use but might be necessary when you start building special-use action queries that need to fill in Parameter values drawn from current or changed versions of the data rows you're working with.

The list in Figure 10.22 shows a partial list of these syntax variations:

Figure 10.22. Syntax variations used to add Parameters to the Parameters collection.

[View full size image]

Note that two lines of code have the dreaded green squiggly disease. In this case, the warning advises that this syntax is no longer in styleit's obsolete, passé, not cool. Microsoft wants you to use the AddWithValue function so the compiler does not have to guess that you're really creating a Parameter with a pre-set value. Actually, except in ASP applications where SqlCommand objects are often used and discarded on each page load, it's unusual to pre-set a Parameter Value property. Generally, this is done "just-in-time" before the SqlCommand is executed. I'll stick with the old tried-and-true methods.

Remember that Parameter objects can be added to the Parameters collection in any orderthey do need unique names and these names must match the parameter placeholders in the CommandText one for one. You aren't permitted to create too many or too few Parameter objects when working with ad hoc queries.

Yes, Parameters can be "gazintas" or "gazoutas"passed to or from the stored procedure. I'll discuss how to set the Direction property later in this section.

Managing ParametersBest Practices

As I've discussed, you should keep in mind that data access is not a constitutional democracy. All input is considered evil until proven innocentespecially when it comes from a user. This means your code is going to have to figure out how, when, and where to make sure the data you pass to SQL Server is pure as the driven snowand it's probably (but not always) best to make those checks before you send the data to the server. While Parameter objects seem to be strongly typed objects, ADO.NET does not care what type of value is assigned to the Value property until the SqlCommand is executed. Basically, the checks your code must perform can be done in several basic ways:

  • Does the value supplied conform to the specified Parameter datatype? That is, are you setting an integer value into an integer type or a string into a string-typed parameter?

  • Is the supplied value's size larger than the capacity of the Parameter datatype? That is, are you passing a 32-bit integer into a 16-bit typed Parameter? While it might be okay to pass smaller-sized values, you won't be able to pass values whose range exceeds the capacity of the target data structure.

  • What about the string values you're assigning to the Parameter Value property? Are they longer than the defined length of the Parameter? Strings are handled a bit differently, as I'll illustrate, but the code behind the scenes[3] won't let you move too much string data into a variable without complaining. Note that you need to set the Size property only for variable-length string Parameter objects.

    [3] Is this code behind the scenes ADO.NET or the .NET Framework, or is it simply the compiler? Does it really make any difference? Not really, but it does make a difference when the validating code does the checks.

  • Is the data formatted to conform to the requirements of the target datatype? For example, when you're working with dates, the strings you supply must conform to one of the recognized date formats. You also need to consider international date-formatting issues.

  • Is the parameter constructed to include special search or wildcard characters? I often have to do last-second concatenation to parameters that pass values into LIKE expressionsare these correct and in the correct place?

Sure, ad hoc queries are pretty tolerant when it comes to the scaling and size of the data you pass as parameters, so you don't have to worry too much about sizes or numeric data ranges. Just make sure your dates and LIKE expressions are correct. However, remember that the data columns in the database tables are all specifically sized one way or anothereven the variable-length columns and TEXT/IMAGE types have upper limits (albeit very large).

Discussion about when, where, and how this validation should be done has been raging for as long as intelligent workstations have existed. Should these tests be done on the user's system (as you can easily do with a client/server application) or in code running in a browser? Should a middle-tier application or the ASP page perform the checks or simply depend on the server, where criteria checks, rules, and complex logic in stored procedures can weed out the trash? All of these questions are for you and your application architect to answer.

Yes, some of these checks are "business rules" that should be isolated and performed on the server so that when (not just if) they change, deployed applications need not be redeployed, but some of these checks simply must be done before you can pass the data to another tier for processing. In this case, you'll need to mask, format, and range-check the input data.

I like to use "pick lists" for as many values as possible. This eliminates the possibility that users will incorrectly enter the values. Pick lists can be loaded very quickly from tables designed for this purpose. These are often the "foreign" key tables such as "ValidStateCodes" or "CurrentDiscounts" that are joined to other tables to ensure data integrity. Lookup tables are usually fairly short, simple lists of codes and descriptions that make it easy for users to choose correct entriesinstead of trying to type them correctly.

In any case, you should eliminate situations where the user is encouraged to type in SQL. As I've said, this is especially dangerous and requires considerable end-user training to accomplish. If you take this approach, it means you'll have to parse every single string provided as "SQL" to ensure that the user does not damage, see, or change anything they're not allowed to view or access. You shouldn't depend on users to enter wildcard ("%") or pattern-matching characters when performing "fuzzy" or ambiguous searches. Of course, when this is necessary, your code should assume responsibility to fill in wildcard characters at the appropriate places.

Regardless of the query approachbut especially in fuzzy searchesbe sure to limit the number of rows returned to keep from overwhelming the system with work and the user with information they can't possibly use. This can be done with the TOP operator or by setting ROWCOUNT in your query, as shown in Figure 10.23.

Figure 10.23. Setting the maximum rows to be returned by using ROWCOUNT or TOP.

Setting the Parameter Value Property

When the time is right and the data is right, you're ready to set the SqlParameter object's Value property.

So what happens when you pass invalid data to a Parameter.Value property? Well, it depends. What do you suppose would happen if I executed the code shown in Figure 10.24? Clearly, there are a lot of errors here, but when is the SqlException handler calledor is it?

Figure 10.24. Discovering when Parameter data failures occurthe hard way.

[View full size image]

It turns out that when you assign junk to the Parameter Value property, nothing happensnothing at all. Even though each and every one of the errors shown in Figure 10.24 can cause the ExecuteReader to abort and throw an Exception or SqlExceptionwhich is sometimes (just sometimes) useful in figuring out what went wrong. So, adding a TRy/Catch around the Value assignment statements will do you no good at all. It seems that ADO.NET blindly accepts anything thrown at it when assigning the Value property. Even when the types are defined correctly but the data is clearly not in type specs, the Value assignment goes through.

None of these Parameter values is questioned at all until the ExecuteReader function is calledyou attempt to execute the TSQL. By then, you have no way of knowing which Parameter is at fault or even if it's an invalid Parameter Value assignment at all. The single exception is also returned by either the SqlException or the general Exception handler, and there is no back reference to the code that actually made the assignmentbut how would it know?

Using the ResetDbType, ResetSQLType Methods

As discussed earlier, when you set the Parameter Value in code, ADO.NET infers the DbType and SqlDbType from the value supplied. In cases where you want to get ADO.NET to reinterpret the Value property, you can call either of these methodsthey both reset the DbType and SqlType based on the Value property.


Validating the Value

Peter and I spent more than a week researching and discussing the best ways to deal with SqlParameter Value validation. The challenges we faced and address in this chapter are:

  • When a new SqlParameter object is instantiated, the DbType and SqlType are set to the specified value in the constructor and match the datatype of the Value (if it's supplied) or NVarChar if it's not supplied. However, since the Value property is often (usually) reset based on user input or by program logic just before the SqlCommand is executed, it's essential that the supplied input variable datatype match the SqlParameter object's assigned datatype. Since the TextBox control (a typical source of user input) manages its Text property as a String, the value can be virtually anything, so you'll need some technique to validate the TextBox Text to ensure it conforms to the correct datatype.

  • Since user-provided values are usually the source of incorrect data, it's essential that input (or edited) data be validated for datatype, range, business rules, and "reasonableness" while the user is focused on the input control. This means that the TextBox, DataGridView, MaskedEdit, ComboBox, or whatever UI control is used to capture input needs to be programmed to accept only valid data. This way, if an error occurs, the user can easily change the data to match the criteria. Later in this chapter, I illustrate ways to add properties to the TextBox control to validate the datatype and ensure that data provided falls within a specified range.

  • Data also needs to be restricted and validated based on a number of other ever-changing criteria, such as minimum and maximum permitted values, precision, scale, and format. These can be considered business criteria, but as often as not, they're simply common-sense rules. For example, while a human's age in years is always a positive value between 0 and 120 (with a few historical or mythical exceptions), if you're working with adults that can fly an airplane, the range is far narrower. It's important to separate the criteria that is likely to change from time to time (more or less frequently) from those rules that really never change. This way, you can more easily adapt the application to evolving business rules without redeploying.

  • Data is often supplied in specially formatted structures. For example, in the U.S., Social Security numbers are formatted as 999-99-9999 and phone numbers are formatted as (999) 999-9999. While these formatting regimens are different in other cultures or business, every application has to work with one formatting schema or another. Data must not only conform to these structures, but also the formatting characters can be stripped or left in place when passed to the Parameter Valueyour datatype must be set to accommodate the chosen approach.

Over the years, the Hitchhiker's Guide has discussed ways to manage business rules of all kinds. One approach that still makes sense uses Extended Properties that are exposed on all SQL Server objects. These properties can define a virtually unlimited set of criteria for individual database objects such as the table, the view, or (more relevant to this discussion) the table's column. For example, you can set Extended Properties to indicate minimum, maximum, suggested, default, mask, description, caption, font, color, or anything else you can imagine. When your application starts, it can easily fetch these properties and use them to tune the business rule logic or the UI in your application.

Writing Code to Validate the Value

Let's take a look at a few techniques that can be used to validate the Value property. The trick here is to choose the "right" time to perform this validation. The approach I settled on[4] validates keystrokes as the data is entered and revalidates the Text property against a selected data type and range. I implemented this by creating a custom data-entry control (inherited from the TextBox control) that monitors keystrokes passed to the control and permits only key codes that apply to the desired data value based on the data type. For example, for number-type data types, I accept only numbers. Since I trap the KeyDown event and filter for numbers, what about the decimal point? If I accept a floating-point number, I need to permit a decimal point. And what if the user makes a mistake and presses the Backspace, Delete, or arrow keys? I need to accept those, too.

[4] See "ValidatingTextBox" on the DVD.

To permit the developer to set the acceptable datatype within a specified range, I added properties to the TextBox control to expose SqlType, Min, and Max properties, as shown in Figure 10.25.

Figure 10.25. Additional properties added to inherited TextBox control.

IMHO

It's best not to trust data from any sourceinternal or external until it's validated. All data is evil until proven innocent.

Since I added these properties to the TextBox control, I can program it to accept (only) data that conforms to structure and scale of the appropriate datatype based on the type of data I expect to capture. For example, if I expect a date, I can set the SQLType property to DateTime or Smalldatetime. Because I'm using the Framework to validate the data, I don't have to write complex Case (or Switch) statements to test for specific types and the corresponding system datatype.

Once I set the SQLType property in my custom TextBox control (which exposes the property as a drop-down list of known properties), I can trap the keystrokes entered in each instance of the control. The routine shown in Figure 10.26 illustrates how I test for acceptable KeyValue values returned from the keyboard. If the key isn't acceptable (based on the datatype set), I tell the control to ignore the keystroke (e.SuppressKeyPress).

Figure 10.26. Validating keystrokes based on the datatype selected.

[View full size image]

Once the user moves the focus from the custom TextBox control, the custom Validating event fires (which overrides the TextBox control's base class Validating event). Here, I further evaluate the string entered by the userensuring that the TextBox Text property is correct based on the datatype selected and falls within the specified range. There is another consideration you need to make when working with TextBox controlswhat happens when the Text property is changed programmatically? Well, the TextBox base class does not fire "Validating" when data is provided by setting the Text property in code, so it's up to your custom TextBox control to handle this.

Based on the SQLType property I've set in the control instance, I call an appropriate TryParse method against the target datatype. The advantage to this approach is that it (should) return Boolean False if the TextBox.Text does not conform to the constraints of the datatype. If TryParse returns True (and does not throw an exception), I know the TextBox Text property is type conformant. Sure, this Validating routine as shown in Figure 10.27 might need a few more Case statements to deal with other datatypesone for each datatype you expect to validate.

Figure 10.27. Validating the Text Property datatype for conformance.

[View full size image]

If the custom Validating TextBox control finds that the TextBox Text property is not set to a value that can be interpreted as the selected datatype, it changes the control's BackColor property and enables the ErrorProvider. The ErrorProvider control is a handy way to show users that there are issues with the data they've entered. It flashes on the form near the control in question and permits you to set a "tooltip"-style message to clarify the problem, as shown in Figure 10.28.

Figure 10.28. The ErrorProvider helps inform the user of data-entry problems.

[View full size image]

The next step in the validation process is range checking, as shown in Figure 10.29. Here the control compares the custom TextBox.Text property against the Min and Max custom properties I added to the control. Again, I use the ErrorProvider and BackColor to indicate a problem. Sure, you might not want to do both, but I kinda like the belt and suspenders approach.

Figure 10.29. Testing the Text property for proper range.

[View full size image]

There is a wealth of third-party and freeware controls you can acquire to provide far more sophisticated validation. Many developers working on more complex projects create a suite of custom controls designed to deal with special-case fields. Some actually auto-generate forms based on the columns returned by a query. By placing this common validation code in custom controls, you can dramatically reduce the complexity of the applications you build. It also makes it easier to build additional applications more quickly and more cheaply. No, I don't suggest you write these controls yourselfnot before searching for third-party or freeware controls that might do everything you need (and probably more). Since these freeware controls are provided with source, you don't have to worry about supportability or security issuesas long as you review the code. If you see an obscure block of code connecting to a strange website, you might want to recode or consider another vendor. Greg Low[5] tells us he's writing an MSDN article on a suite of controls he's written. I suggest you look for it along with other sources, including Billy Hollis's validator control samples[6]. Both of these gentlemen provided guidance as I wrote this chapter.

[5] Greg Low wants folks to find him through his blog at http://msmvps.com/greglow

[6] I have included Billy's validator controls on the DVD.

Handling Complex Parameter Design Issues

There are a few issues that I need to discuss before I'm finished touring the Parameters collection and executing dynamic queries.

Working with Unicode Literals

If you get Visual Studio or SQL Management Studio to generate code for you, you'll doubtless see your string literals prefixed with an "N". This simply means that the string literal is to be treated as a Unicode value and stores 16 bits per character. This approach is essential when working with Unicode character sets used by a host of languages. Note that Unicode-typed columns (and parameters) consume twice as much space as their ANSI (8 bits per character) equivalents.

Duplicate Parameter Names

If you execute a multi-part query, you might find that the parameters passed to the various queries and (possibly) action commands have the same parameter names. For example, if your code executes the same stored procedure more than once, you'll face this problem. Sure, it's a great idea to bundle many operations in the same batch, but this won't work if the Parameters collection can't isolate the individual SqlParameter objects by name. Since the parameter names have to match the names in the stored procedure, there is no work-around besides breaking up the operations into separate batches.

Ensuring Your Command Rights

Regardless of the approach you use to execute your T-SQL, the query you execute requires that your application's credentials (whether inherited from the current user or hard-coded) are sufficient to execute the query against the specified table(s) and no more. These access rights can extend right down to the table column. You should never grant a user more rights than they absolutely need. If you take this approach and your code or application is penetrated, the damage can be limited to a far smaller surface area. That is, if the application credentials permit only read-only access to a specific set of tables or (better yet) views, the malicious act can do only what the application was designed to do in the first placereturn rows from these read-only areas; it can't change anything anywhere in the database. If you need read-write access, plan to use stored procedures to perform these changes. That way, the hacker can change the database only via carefully protected stored procedures. When you code action commands, another world of issues is exposed. I'll talk about update queries in Chapter 12.

Handling Nulls

Every SQL Server developer will learn at one point in time or another how to handle a data vacuum. That is, when you don't have a value to apply to a database column you're forced into a quandarycreate a tag that means "data is not (yet) known" or use NULL. In the 1970s (and before), database developers were faced with this problem, but 80-column cards and paper tape didn't really have a special hole for the absence of data. If a column were left blank, it was coded as an EBCDIC 0x20so it was stored as a space. I made do with that until disk- and tape-based systems encouraged us to come up with something better. In a relational database, I use the concept of NULL to indicate that the data is simply not known or unavailable. This raises a number of issues. How does NULL average in with other data? Can one compare a "dunno" with a valid entry? I'll leave these questions up to the books on computer science. I won't get into a long-winded discussion about the wisdom of using NULLs, but I do think NULL is a better placeholder than BHOM (Beats the Hell Out of Me), which Bill used when he taught COBOL programming for the U.S. Army.

In those situations where you need to pass a NULL to a SqlParameter or test a data value for NULL, you'll have to step off the boards a bit and use a number of special notations to let ADO.NET know what you want done. These tasks are complicated somewhat with the variations in Visual Basic .NET and C#, and I'll touch on those as well.

Testing for NULL Values in Visual Basic .NET

Visual Basic .NET (and the .NET Framework CLR languages in general) supports many techniques for handling NULL values, as the following code examples show. Figure 10.30 starts the exercise by creating a simple SqlDataReader that returns a NULL value for the Year_Born column. Notice how I set the SqlParameter Value to SqlTypes.SqlInt32.Null. Each of the SqlTypes exposes a Null property that can be used to generate a type-correct NULL value.

Figure 10.30. Fetching a DataReader from a query that returns NULL for the Year_Born column.

[View full size image]

At this point, if you try to reference the value returned in the DataTable rows, you'll discover that the CLR types don't support NULL type conversion. This means your code throws an exception as you try to retrieve a value from a DataTable or DataReader rowset that's set to NULL. This means you must add code for each and every column you expect to contain NULL to test for this contingency. Figure 10.31 illustrates several techniques you can use to test for a NULL column value. Note that you can't use the "=" operator because you can't "compare" nothing with anything.

Figure 10.31. Testing for a NULL column value.

[View full size image]

Using Dynamic SQL

Another (far more dangerous) approach to difficult parameter query issues is to pass the components of the query to the server to be compiled in situ. That is, your CommandText can call a stored procedure that takes the inbound parameters and constructs an entirely new SQL query and uses the T-SQL EXECUTE (EXEC) operator to compile and execute it. In this case, you can pass any or all of the T-SQL you want to execute as parameters, assemble it on SQL Server using string concatenation, and use the EXEC operator to compile and execute the code. Let's attack my IN expression challenge with the EXEC approach. Figure 10.32 illustrates how to code a stored procedure that accepts the list of acceptable prices as a delimited string. In this case, the entire compile phase is repeated each time the stored procedure is executed. This approach also permits users to (easily) inject SQL via the single @PricesWanted string parameterwhich is why this approach is not recommended.

Figure 10.32. Using dynamic SQL and the T-SQL EXECUTE operator.

IMHO

Using dynamic SQL can make SQL injection a lot easier. At least the hackers will thank you.

Understanding SqlCommand Events

There are only two events exposed on the Command objectDisposed and StatementCompleted. I'll be talking about both of these Events in Chapter 11 when I discuss executing SqlCommand objects.


Executing Stored Procedures

I won't bore you with another lecture on the importance of stored proceduresyou've heard it a dozen times or more in this and earlier chapters, and from other folks in the industry that we both respect. Suffice it to say that if you're using SQL Server, I expect you to use stored procedures at one point in time or another. When setting up a SqlCommand to execute a stored procedure, you simply set the CommandType to "Stored Procedure" and build a Parameters collection to match the stored proceduredefined parameters. The names you use must match the names used in the stored procedure, and you must define and provide a Value setting for each stored procedure parameter that does not have a default value assigned in the stored procedure. That much should be clear by now. What I haven't talked about is the Direction property of the SqlParameter object.

Handling OUTPUT, INPUT/OUTPUT, and RETURN Value Parameters

All stored procedures returna RETURN value. While the stored procedure code might set the RETURN value, a value of 0 is sent nonetheless if no specific RETURN value is set. In order to trap this value, you'll need to add a SqlParameter to the SqlCommand Parameters collection and set the Direction property to ParameterDirection.ReturnValue. The name of this SqlParameter is not important, but the SqlDbType isit must be set to SqlDbType.Int. The RETURN value can be used for virtually anything that can be expressed as an Integer, but it usually indicates the success or failure of the stored procedure.

If the stored procedure returns OUTPUT parameters, you must create a SqlParameter object to capture the valueone for each OUTPUT parameter. No, these are not optionalif a stored procedure defines a SqlParameter as OUTPUT, you must define a SqlParameter with the Direction property set to ParameterDirection.Output. OUTPUT parameters can return any type of data except image, text, and ntext (except when working with CLR stored procedures).

If you want to use the same SqlParameter to pass a parameter value to and from a stored procedure OUTPUT parameter, you simply set the Direction property to ParameterDirection.InputOutput. In this case, you must provide a value for the SqlParameter before execution. Figure 10.33 shows a stored procedure used to demonstrate use of the various Parameter types. Note that this stored procedure returns OUTPUT and RETURN value parameters but no rowset.

Figure 10.33. The TestInOutNoRowset stored procedure.

The code shown in Figure 10.34 illustrates how to set up a SqlCommand to execute a stored procedure, set the input and input/OUTPUT parameters, and extract the OUTPUT and RETURN value parameters. I use the ExecuteNonQuery to execute the stored procedure. Because the stored procedure does not return a rowset, the RETURN value and OUTPUT parameters are available immediately. Remember, OUTPUT and RETURN parameter values are not returned until the last rowset is returned from the query.

Figure 10.34. Configuring a SqlCommand to execute the TestInOutNoRowset stored procedure.

[View full size image]

If the stored procedure returns one or more rowsets, you'll have to capture or cancel all of the resultsets before any OUTPUT or RETURN value parameters are exposed. This means the SqlParameter Value property for the OUTPUT and RETURN value parameters won't contain meaningful information until all rowsets have been returned.

Handling Rowsets with OUTPUT Parameters

Before I move on, let's take a closer look at a fairly common scenariowhen a stored procedure returns two or more resultsets along with the OUTPUT parameters. The example in Figure 10.36 returns two resultsetseach with a rowsetand then returns two OUTPUT parameters. Before you peruse the code, take a look at the stored procedure (shown in Figure 10.35). Note how the each OUTPUT parameter captures the @@Rowcount value after the SELECT statement is executed. By this, one might assume that the first OUTPUT parameter would be made available directly after the first rowset is processedit's not. All OUTPUT parameters are returned by SQL Server after all resultsets and their rowsets (if any) are returned.

Figure 10.35. Return two resultsets containing rowsets with OUTPUT parameters.

[View full size image]

IMHO

OUTPUT parameters are one of the most under-used and most misunderstood TSQL mechanisms.

So the code to execute this stored procedure uses the DataSet Load method to retrieve all resultsets and move any rowsets into the named DataTablesin this case, Load creates two new DataTables. The Load method is also doing something else for usit's closing the SqlDataReader when it completes rowset population. Once the SqlDataReader is closed, you'll be able to see the OUTPUT and RETURN value parameters. They won't be visible until theneven if the Read method (that I discuss in the next chapter) returns False (there are no more rows).

Figure 10.36. Executing a stored procedure that returns two resultsets and OUTPUT parameters.

[View full size image]


Summary

I expect that the SqlCommand class is the most referenced of all of the ADO.NET classesit's the only way you can execute a query with ADO.NET. Clearly, the SqlCommand class is your only query portal to SQL Serverit's used to format the queries you executethe questions you want to ask of SQL Server. It's also used by the Visual Studio tools to generate DataAdapter and TableAdapter objects, as well as in code you pound out on the keyboard. SqlCommand objects are also generated by the SqlCommandBuilder class both by the Visual Studio tools and (in some cases) by your code to generate the action queries needed to update your database. As you saw, the Parameters collection plays a big role in how you insert user input into these queries to focus the SELECT statements and populate the columns inserted into the database. The SqlCommand is also a gateway for hackers. Because of this, you need to ensure that all values captured from external sources are considered to be untrusted until validated. In later chapters, you'll see how to execute the SqlCommand object to return rowsets and make changes to the database.


Chapter 11. Executing SqlCommand Objects

    Introduction

    Executing SqlCommand Objects Synchronously

    Understanding the SqlDataReader

    Populating Data Structures

    Fetching Rows Asynchronously

    Executing Commands Asynchronously

    Summary


Introduction

Once you've created a SqlCommand object and populated the Parameters collection, you're ready to ask SQL Server to execute the T-SQL it contains. In this chapter, I discuss how a SqlCommand can be executed either synchronously or asynchronously. I'll show you how to choose what kind of information you want returned to your applicationa rowset, a scalar[1], the rows affected, or an XML stream. You can also choose whether or not to return additional keyset or schema information from a querywhich can be handy if you're building tools or data-driven routines. Choosing the "right" execute function is really pretty importantif you choose the wrong one, you can waste considerable time and resources processing data that you'll never use (or be able to use), so I make sure you know how to choose the right tool for the job.

[1] A "scalar" is simply a single value returned as an object.

Regardless of what your code executes, the login account you use must have sufficient (at least, read-only) rights to execute the query. If you plan to update the data, you'll need read/write rights. While this might seem obvious, if your DBA has protected the database (as he or she should), you won't be granted rights to any base tables at all and only limited rights to the specific stored procedures you need to run the application and perform any needed updates.

The primary purpose of a SqlCommand object is to provide a mechanism to execute the CommandText and get the server to do something (like an action command) or return one or more resultsets and the rowsets they contain. ADO.NET 2.0 does not expand on the list of synchronous SqlCommand execute functions, but it does add an entire suite of asynchronous functions to execute your queries. These new functions can dramatically alter the way you approach query problems and help deal with queries that take an inordinate length of time to execute (at least, as far as users are concerned). First, I'll deal with the synchronous execute functions as listed in Table 11.1. Later in this chapter, I'll discuss the new asynchronous methods.

Table 11.1. The SqlCommand Synchronous Functions

Execute Function

Returns

Description

ExecuteNonQuery

Integer

Executes SqlCommand CommandText and returns the rows affected integer

ExecuteReader

SqlDataReader

Executes SqlCommand CommandText and returns SqlDataReader (two overloads)

ExecuteScalar

Object

Executes SqlCommand CommandText and returns first column from first row from first rowset returned from query or Null

ExecuteXmlReader

XmlReader

Executes SqlCommand CommandText and returns System.Xml.XmlReader


Executing SqlCommand Objects Synchronously

All of the synchronous SqlCommand execute functions "block" as soon as they are executed. That is, before the server returns the first indication that a resultset is ready to process, the thread and connection used to execute the function are prevented from proceedingthey're blocked. This means you need to be prepared to wait for, at most, CommandTimeout[2] seconds before another line of code on the thread that executed the function can proceedyour user has to wait, too. You can't depend on a Timer control or other events (on the same thread) to interrupt this processyou don't own the thread because it's being used by ADO.NET. This means you can't increment a progress bar or other animation to entertain your users and prevent them from pressing Ctrl+Alt+Delete or switching to another page. You also won't have an opportunity to execute a Cancel method if the query runs too longthat is, not unless you fire up another thread (like a BackgroundWorker thread) to do so.

[2] I discussed the CommandTimeout property in Chapter 10, "Managing SqlCommand Objects"remember, the time you wait can be far longer than this setting.

Yes, if you're so inclined, you can start another thread of execution to execute code while the SqlCommand execute function is running. But what's the point? If you use the asynchronous versions of the execute functions, you won't have to. But what if you need to simply cancel a longer-than-anticipated execution? If this is what you have in mind, you can indeed use a BackgroundWorker thread to execute the Cancel method, but it won't be easy. Suffice it to say, while the execute function is running, nothing else on your thread of execution can get any attentionnot until the server returns the first resultset. As I'll show you later, it's even more complicated than that....

Let's take a closer look at each of the execute functions. Table 11.1 lists the four synchronous execute functions, and shows the object type returned.

Choosing the "Right" Execute Function

As with most performance and architecture tips, I suggest you use the execute function that does what you want and no more. For example, if you need to execute only a DML command (an action command), don't use an execute function that expects a rowset. If you need a single value (like a count or a Boolean) from the server, use the ExecuteScalarnot ExecuteReader or ExecuteXmlReader, as they are overkill. Consider that rowset processing is complex and expensive, so you should take as many steps as practical to avoid data access strategies that depend on rowsets. Returning XML is the most expensive approach of all. While it's useful to communicate with disparate data sources and XML web services (and space aliens), it's usually too verbose and inefficient to use except when it's really needed. Whenever, possible, use a rowset-less execute function with parameters. You'll find that it's far faster to return 100 OUTPUT parameters from SQL Server than a single row of data. Table 11.2 categorizes the various query types and the best execute function to use.

Table 11.2. Choosing the Right Execute Function by Task

Task

Best Execute Function

Execute an action or DML command, such as UPDATE, INSERT, or DELETE and capture rows affected value

ExecuteNonQuery

Return value from stored procedure OUTPUT or RETURN value parameters but no rowset

ExecuteNonQuery

Return string or other value from query but no rowset

ExecuteScalar

Return rowset and (optionally) RETURN value and OUTPUT parameters

ExecuteReader

Return an XML data stream

ExecuteXmlReader

IMHO

Returning XML is the most expensive approach of allbut particularly effective when communicating with space aliens.

Managing the Connection

All SqlCommand execute functions require an open SqlConnection. Your code must hold it open until all rows are fetched. That is, the SqlConnection object associated with the SqlCommand must be open before execution and must be left open until any rowsets are fetched or you execute Cancel. This means if you pass the SqlDataReader to another function, it must (somehow) Close (or Dispose) the SqlConnection used to feed the SqlDataReader. If it does not, you risk the (very real) chance of orphaning the SqlConnection object and (eventually or possibly very quickly) overflowing the connection pool.

To make SqlConnection object management easier, the ExecuteReader function can be programmed to automatically close the associated SqlConnection as soon as the SqlDataReader itself is closed. The SqlDataReader supports a Close method, which is called:

  • Automatically when you pass (bind) a SqlDataReader to a complex bound control (one that accepts an entire rowset, like the DataGridView, ListBox, or ComboBox controls).

  • When you use the DataTable or DataSet Load methods. Only the ExecuteReader function has this featurenone of the other execute functions supports automatically closing the connection when the SqlDataReader is closed, so you'll need to be especially vigilant when working with execute functions other than ExecuteReader.

    Tip

    Only the ExecuteReader supports the CommandBehavior.CloseConnection option.

  • When you execute the Dispose method on the SqlDataReader. No, there is no compelling reason to call Dispose on a SqlDataReader, other than to take advantage of the underlying Close it performs.

  • When you use the Using syntax to declare the SqlDataReader. Figure 11.1 illustrates the use of the Using operator to manage the SqlConnection and SqlDataReader object instantiation state. When the logic reaches the End Using, the object Dispose method is called behind the scenes. (The Using operator is a Visual Basic .NET 2.0 Framework feature). This approach is appropriate when you create a just-in-time SqlConnection object in a function and don't intend to share it or use it outside the current code scope.

    Figure 11.1. Using the Using operator to manage object state and tear-down.

    [View full size image]

C# developers have worked with the Using block since its inceptionit was added to Visual Basic .NET in Visual Studio 2005. Using guarantees that the object declared by the Using operator is disposed of when the code path leaves the Using/End Using block. This means that if there is an exception (even an unhandled exception), the object is properly disposed of. It also means that objects created within a Using block can't be passed outside the block.

Using ExecuteNonQuery

The ExecuteNonQuery function is the fastest (and lightest) of all of the SqlCommand functions, so it's understandable why it has the fewest features. It's used to execute action queries or any time when you don't care about or don't expect a rowset to be returned from the query. If a rowset arrives, it's ignored. However, any OUTPUT or RETURN value stored procedure parameters are returnedimmediately, as there is no rowset to return.

Capturing the "Records Affected" Value After Any Command

ExecuteNonQuery can return an integer (bigint) that's set by the records affected[3] value returned by the queryunless you turn it off. One common practice is to use the SET NOCOUNT ON phrase in your T-SQL. This tells SQL Server to skip the extra packets needed to report rowset-less resultsets and the rows affected value. It makes handling multiple resultset queries easier and reduces the amount of unneeded information on the wire, but it disables the ability to trap the (often useful) rows affected value. It turns out that the ExecuteReader automatically ignores all resultsets that aren't bearing rowsets.

[3] a.k.a. "rows affected".

The example shown in Figure 11.2 illustrates setting up an action command that performs an UPDATE on the database but does so in a transaction, so I don't arbitrarily change the data. In this case, the rows affected value is returned by the ExecuteNonQuery function. Yes, even SELECT statements return a rows affected valuethe SqlDataReader.RecordsAffected property exposes this integer.

Figure 11.2. Using ExecuteNonQuery to execute an action command.

[View full size image]

Keep in mind that there is no automatic mechanism provided with the ExecuteNonQuery function to close the SqlConnection after the ExecuteNonQuery is executed. This means it's always up to your code to do sounless you're working with a persistent connection that's left open for the life of the application.

Using ExecuteScalar

ExecuteScalar is very similar to ExecuteNonQuery, except that it returns the first column from the first row of the first resultset. All other rows and resultsets are discarded after it is executed. Figure 11.3 illustrates the ExecuteScalar function to return an Integer from the query. Note that ExecuteScalar returns an Object, so you'll have to cast it to your variable.

Figure 11.3. Using ExecuteScalar to return an integer.

[View full size image]

Using ExecuteXMLReader

I originally planned to devote a chapter to XML handling, but I ran out of timeI hope to write an EBook on XML handling sometime in the future. For now, consider that since SQL Server 2000 incorporated XML data streams, developers have found ways to use (and abuse) XML. When you execute a T-SQL command that returns XML (using the FOR XML operator), you'll need to capture and manage the returning XML stream. The ExecuteXMLReader function returns a System.Xml.XmlReader object to capture it. Microsoft seems to think this is a good use of your bandwidthespecially for "mass quantities of data". I get an image of the Coneheads consuming mass quantities of beer by the six-packbut perhaps you were too young to stay up for SNL.

Using ExecuteReader

The ExecuteReader is probably the most commonly used SqlCommand execution function so I'll spend quite a bit of time discussing it. It's used when you need to execute any SqlCommand that returns one or more resultsets (possibly) containing rowsets. Of course, you might expect that the query always returns a rowset, but what if it doesn't? Why would executing a stored procedure that usually returns a rowset suddenly not work any longer? Well, what if the stored procedure logic is altered or simply branches into other logic that executes another SELECT or action command before it returns the usual rowset? I'll discuss these issues later in this chapter, but I'll give you a hintmultiple resultsets.

One set of options I want to mention at this point is the SqlDataReader "Command Behaviors" associated with (just) the ExecuteReader function. These options, which can be used alone or in combination, configure the SqlDataReader to enable automatic handling of the data and other functions.

Table 11.3. The ExecuteReader CommandBehavior Options

CommandBehavior

Description

CloseConnection

Closes the associated Connection when the SqlDataReader is closed

Default

Doesn't apply any special options (the default behavior)

KeyInfo

Also returns column and primary key information (with rowset)

SchemaOnly

Fetches column information (only)no data or PK information

SequentialAccess

Loads data as a stream so BLOB types can be fetched incrementally

SingleResult

Returns only one resultsetall subsequent resultsets are ignored

SingleRow

Returns the first row from each resultsetall other rows are discarded

CommandBehavior.CloseConnection

This is one of the most useful CommandBehavior settings. It simply tells ADO.NET to close the connection associated with the SqlCommand when the SqlDataReader is closed. That's great as long as your code or a complex bound control actually closes the SqlDataReader. No, you can't use this option on all SqlDataReader streams you build, as you might want to reuse the connection for other operations after the SqlDataReader is closed.

The problem with this setting is that it assumes that the SqlDataReader is indeed closed somewhere along the line. If you pass a connected SqlDataReader to another layer (all SqlDataReader streams are connected), you might not be assured that the receiving function will indeed close it. If the SqlConnection stream is created in a Private routine, the calling routine cannot address itwhen the creating routine loses scope, its objects are subject to disposal.

CommandBehavior.KeyInfo and CommandBehavior.SchemaOnly

Microsoft wanted to keep the overhead involved with a SqlDataReader as light as possible. To this end, they eliminated the base table name, key column information, and other DDL info from the base structure. If you're writing a tool that needs to examine the schema, you can tell the ExecuteReader method to include it by setting the CommandBehavior.KeyInfo switch. If all you need is schema and no data, you can set the CommandBehavior.SchemaOnly switch. If you want to browse this schema data, you can Or the two options together and use the GetSchemaTable function on the SqlDataReader. Figure 11.4 [4] illustrates using the KeyInfo and SchemaOnly CommandBehavior options to set up a call to the SqlDataReader GetSchemaTable function.

[4] See the "GetSchemaTable" example on the DVD.

Figure 11.4. Using the KeyInfo and SchemaOnly ExecuteReader options to fetch table schema.

[View full size image]

The GetSchemaTable function returns a DataTable that contains the data structure (the schema) of the database table referenced in the SELECT executed by the SqlCommand, as shown in Figure 11.5.

Figure 11.5. The DataTable returned by GetSchemaTable for the Authors table.

[View full size image]

CommandBehavior.SequentialAccess

The wisdom (or lack thereof) of storing Binary Large Objects (BLOBs) like TEXT, IMAGE[5], varchar(max), and varbinary(max) in the database has been hotly debated over the years. Sure, SQL Server has improved the performance and flexibility of these types, but I still find that it's often (far) faster to store this type of data in separate files and use the database to simply index the data.

[5] I've heard that the IMAGE and TEXT datatypes are to be removed from SQL Server. I'll believe it when I see it.

If you're working with a database that includes BLOBs for some reason, you'll probably want to retrieve these values at one point in time or another. The SqlDataReader is capable of extruding a stream containing BLOBs, but it's best to inform ADO.NET that you expect to be processing these long (very long) streams of binary data. By default, the ExecuteReader function starts loading data as soon as the first row is available. Because each row can (potentially) contain over 2GB of data, it's best to fetch the data as it's made available instead of waiting for each row to populate. By setting the CommandBehavior.SequentialAccess option, the ExecuteReader function enables this behavior.

Once you choose the CommandBehavior.SequentialAccess option, a number of new "rules" apply. For example, you must access each field sequentially in your code. For example, if you return four columns and a BLOB, you should access the non-BLOB columns firstin sequence. I think you'll find that there are a number of irritating issues as you work with BLOBs stored in the database. I touch on these in a number of places elsewhere.

CommandBehavior.SingleResult and CommandBehavior.SingleRow

When you need only a single resultset or a single row, you can tell ADO.NET to ignore all but the first resultset or row. Of course, it might be wiser to request only a single resultset or row in the first place by rewriting your SQL or by using a TOP expression. I expect that ADO.NET does not "cancel" the remaining resultsets or rows, but simply fetches and discards them behind the scenes. This can negate any benefit from using these CommandBehavior options.

Note that the higher-level functions in ADO.NET, such as the DataAdapter or TableAdapter Fill and Update methods, also use the low-level ExecuteReader and ExecuteNonQuery functions to perform their operations.

Since you're holding a connection open during rowset population, it's not a good idea to let the user intervene. Rowset population (the process of fetching the rowset) should be completed as quickly as possible so that the connection can be released to the connection pool. This means you should re-examine designs that process inbound SqlDataReader rows as they arrive or let the user decide what to do with individual rows. In most cases, this approach can be better implemented by using server-side logic to process rows.


Understanding the SqlDataReader

If you need to return and manage a rowset returned from the ExecuteReader function (since that's the only way you can fetch a rowset), you have several choices. You can either spend the afternoon writing code to pick off and hard-cast each and every column that's returned and moving the data column by column to where ever you need it (like arrays or individual display controls), or you can take the fast and easy approachuse the DataTable or DataSet Load methods. I leave that discussion to sections later in this chapter that explain and show how to handle resultsets efficiently.

As you saw in Table 11.1, the ExecuteReader function returns a SqlDataReader. It should be more or less familiar to you now, as I've created SqlDataReader streams any number of times in earlier examples. Basically, the SqlDataReader is a "connected" data streamit cannot exist without an open connection to feed it data. It's also a "one-way" stream that exposes only a single row of data at a time. Yes, behind the scenes, the stream is buffered, and you can adjust that size using the PacketSize keyword in the ConnectionString. It's read-only, in that there is no mechanism to change the server-side data. While the SqlDataReader rows are being fetched, the Connection cannot be used for other workunless you enable MARS. The high(er)-level DataAdapter and TableAdapter Fill methods create SqlDataReader streams to fetch rows behind the scenes.

Frankly, I don't usually recommend use of ExecuteReader and the SqlDataReader for many applicationsat least, I didn't. That's because I usually need the features and flexibility, methods and events associated with updateable or simply scrollable and bindable DataTable and DataSet objects. A SqlDataReader data stream in itself is not particularly usefulespecially in Windows Forms applications. While you can bind a SqlDataReader to a bound control in an ASP application, that's not possible in Windows Forms. The SqlDataReader can't be sorted or filtered, and there are no "find" methods exposedbeyond the sort, filter, and find done by the initial query. Remember, the SqlDataReader is a "stream"like a flow of boxes on a high-speed conveyor belt. You can't reorder or filter out selected boxes after they're on the belt. Sure, you can choose the order in which the rows are placed in the stream and even filter in/out those specific rows you want to return, but all of these ordering and filtering operations are done on the servernot while the rows are being transported to your application.

Fetching Data from the Stream

The task of extracting the data returned by a SqlDataReader can be code-intensive and very brittle, as it can be easily broken when you (or someone you aren't speaking to since that incident in the cafeteria) make schema changes. While it's easier than wading hip-deep into freezing water to scoop out salmon from a fast-running mountain stream, it can be just as time-consuming to write the codeat least you don't have to compete with grizzly bears and your butt does not get as cold. Later in this chapter, I'll show you how to manage rowset population with far less code.

Testing the Stream for Data

The SqlDataReader is unlike anything you've used before (unless you're familiar with DB-Library). To start with, you should ask ADO.NET if any rows were returned by the query. Since ADO.NET 1.1, this is accomplished quite easily by using the SqlDataReader HasRows function. If HasRows returns true, there is a populated rowset available on the SqlDataReader. If the query succeeds but returns no rows, HasRows returns false. Since the SqlDataReader steps over resultsets generated by action commands (like UPDATE), you'll get only a RecordsAffected setting (one for each resultset). Figure 11.6 illustrates use of several ExecuteReader functions to execute and process a T-SQL batch, including HasRows to determine whether there is a rowset to process, Read to fetch each row, GetValue to fetch the column data values, and NextResult to step from one T-SQL command to the next.

Figure 11.6. Executing and processing a batch of T-SQL statements using ExecuteReader.

[View full size image]

Unlike ADO classic, ADO.NET handles manual rowset processing very differently. Since the SqlDataReader is a stream, there's no need for any "move" methods like MoveFirst, MoveNext, or MoveLast. These ADO classic Recordset methods made more sense for the ADO classic "cursor"-based approach.

In contrast, the SqlDataReader uses a new way to fetch the data from the data streamwell, it's new to ADO classic developerswe used this approach for years with ISAM data engines and DBLib. In this case, you first test for an available rowset using the HasRows functionif it's true, you can proceed to the Read function, as shown in Figure 11.7. Read fetches the row from the cache and exposes it to the Get methods. Read returns false when there are no more rows to fetch in the current resultset. That does not mean there isn't more data in the stream. To see if there are more resultsets, you need to execute NextResult to position to the next resultset. If NextResult returns true, you need to loop back, test for another rowset using HasRows, and proceed through the Read phase again. Once NextResult returns false, there are no more resultsetsbut there's still more data to fetch if you executed a stored procedure (especially one with OUTPUT parameters). Once the DataReader is closed (using Close or Load or by binding to a complex control), you can fetch the OUTPUT parameters. Now, there's no more data to fetchexcept the RecordsAffected property. Having fun yet? Of course, the Load and Fill functions do all of this for you with a single line of code.

Figure 11.7. Processing the SqlDataReader stream.

You'll also see the Read method reappear if you start working with the SqlCeResultSet cursor, which exposes a scrollable cursor that uses Read, ReadNext, ReadPrevious and ReadLast to position the current row pointer. Be sure to read my new EBook Hitchhiker's Guide to SQL Server Everywhere[6]it explains the SqlCeResultSet as well as a myriad of other SSEv details.

[6] Visit the Hitchhiker's Guide support site for more details. www.hitchhikerguides.net

Fetching Individual Rows

Since the SqlDataReader is just a stream of rows being returned by the interface, it does not have an "in-memory" representation. Until the 2.0 Framework, this means that you have to (manually) move the rows and columns (one by one) into variables, an array, or a control of some kind to be able to use them. As I've said before, the new DataTable Load method virtually eliminates the need to do this yourselfat least, for simple (single) resultsets. ADO.NET 2.0 permits you to create a SqlDataReader stream and use the Load method to read the first rowset into a new DataTable. Assuming you can work with a DataTable and can afford the slight overhead associated with a DataTable, this approach makes a lot of sense. While the DataTable is a bit more expensive than a simple array, it exposes a wealth of features that you're likely to use. There are a number of other benefits to the Load method. For instance, you don't have to create a DataAdapter, a separate SelectCommand, and use the Fill method when you simply want to populate a DataTable from a query. You also don't have to hard-code the DataTable definition or generate it with some clever code that constructs it based on the SqlDataReader schema.

Yes, there are times when you're going to need to extract the data from a SqlDataReader manually. Let's walk through these techniquesif for no other reason than to show you how much trouble you'll be faced with when you don't use the Fill or Load methods to extract SqlDataReader data. After you've executed Read, you can proceed to fetching the data values from the stream. Don't forget this step. If you do, you'll throw a "Data not available" exception.

In the first version of ADO.NET (1.0), there was no GetRows function, so developers had to execute the Read method to determine if one or more rows were returned by the query. Unfortunately, Read also moves the "current row" point ahead, so if you use Read to test for rows and then bound the SqlDataReader to a complex bound control, the first row would be lost. That's because bound controls automatically execute a Read function before extracting data from the SqlDataReader.

Tip

Don't forget to call Read before using the Get functions.

Let's start with an example (as shown in Table 11.1) that calls the same MultipleResultsetOUTPUTTitlesPriceByState stored procedure as I called in the previous chapter. In this case, I don't (can't) use the DataTable Load method because it closes the SqlDataReader as soon as it processes the first SqlDataReader. This prevents one from capturing the rows from the second or any subsequent resultsets.

Figure 11.8. Call a stored procedure that returns multiple resultsets.

[View full size image]

Instead of using the DataTable Load method, I use my own custom class GetTable, as shown in Figure 11.9. While not particularly efficient, it illustrates the code needed to access specific column values. This routine first uses the SqlDataReader GetSchemaTable method to return the SqlDataReader schema and builds a new DataTable Columns collection from this DDL definition. Once the DataTable is configured to match the schema of the inbound SqlDataReader, the data is extracted in the example row by row and column by column using the SqlDataReader GetValue method in a couple of loops. This means that if the schema changes (as it often does as your database matures), this code will have to be recoded and redeployed. I'll discuss the other "Get" methods next and more efficient methods introduced in ADO.NET 2.0 later in this chapter.

Figure 11.9. Create and populate a DataTable from a SqlDataReader.

[View full size image]

Using the SqlDataReader Get Functions

There are a wealth of "Get" functions associated with the SqlDataReader used to retrieve the schema and metadata, as well as fetch data from selected columns in the data streambasically, there is a Get function for each of the supported datatypes, plus a couple to return object values. Remember that ADO.NET returns data as an Object, so you'll want to cast the values into type-specific variables as it's imported into your application. This is what the Visual Studiogenerated strongly typed TableAdapter and DataSet classes do.

IMHO

Managers appreciate it when developers write fewer lines of code. These apps are cheaper to develop and easier to support.

As I'll show you, you can use the GetValue function to extract data without specifying the datatype, but this is (marginally) more expensive than fetching data with type-specific Get functions such as GetInt16. Table 11.4 lists and explains the basic characteristics of these methods.

Table 11.4. The Type-Specific Single-Column SqlDataReader Data Extraction "Get" Methods

SqlDataReader Get Function

DescriptionFetches Column Data as a...

GetBoolean

Boolean

GetByte, GetBytes

Byte or "N" bytes into a Byte array

GetChars

"N" characters into a Char array (GetChar is not supported in SqlClient)

GetDateTime

A date as System.DateTime object

GetDecimal

Decimal

GetDouble

Double

GetFloat

Single

GetGuid

System.Guid

GetHashCode

Integer

GetInt16

Short

GetInt32

Integer

GetInt64

Long

GetString

String

GetValue

System.Object (exposes the underlying variant)

GetProviderSpecificValue

System.Object represents provider-specific value

GetSqlXml

XML stream

Don't get me wrong. I think that there are precious few situations where you should use these functions. First, they are code-intensive. If you take this approach, you'll write and have to support a dozen lines of code (or perhaps far more) to simulate part of the functionality exposed in the Load or Fill methods. If your schema changes, you'll be back recoding and redeploying your application. Yes, there are situations where you want to fetch only one or a few columns from a SqlDataReader stream, and perhaps (just perhaps) this approach makes sense.

There is another entire set of these Get functions that is used to return SqlTypes. I won't list these here, as they're pretty intuitive. For example, GetSqlString returns a SqlTypes.SqlString. These are useful when targeting SqlType variables.

Fetching All Columns

There is another approach to fetch the data columns that I should mention: GetValues and GetProviderSpecificValues. These functions move all of the columns in the SqlDataReader row into an array of type System.Object. Okay, now that the row is loaded into an array, what can you do with it? I discussed binding to arrays in Chapter 8, "Getting Started with ADO.NET." Again, I don't see the real utility herewhy not just create a DataTable and populate it with Fill or Load?

Fetching Metadata from the SqlDataReader Stream

Table 11.5 lists the SqlDataReader metadata functions used to return detailed information about the SqlDataReader and the data it's exposing.

Table 11.5. The Type-Specific Single-Column SqlDataReader Data Extraction "Get" Methods

SqlDataReader Metadata Function

DescriptionReturns...

GetDataTypeName

Source datatype name (such as bit, char, datetime...), given column ordinal.

GetFieldType

System.Type of column (such as System.Boolean, System.String, System.DateTime), given column ordinal.

GetName

Name of column, given column ordinal.

GetOrdinal

Column ordinal (zero-based), given the column name.

GetProviderSpecificFieldType

System.Data.SqlTypes of specific column (such as SqlBoolean, SqlString, SqlMoney).

GetSchemaTable

System.Table containing metadata from stream (see discussion in Chapter 8).

IsDBNull

Boolean indicating whether the column is NULL by column ordinal. Available only post-Read.

Implementing these Get functions (as illustrated in Figure 11.10) is straightforward. All of them can be called prior to Read except for IsDBNull. These functions are critical to any code trying to manually decode the SqlDataReader stream.

Figure 11.10. Implementing the SqlDataReader to get metadata functions.

[View full size image]

As I'll illustrate when I start discussing the new TableAdapter class and the "old fashioned" SqlDataAdapter, there are other ways to execute queries that return rowsets like the Fill function. I'll discuss these later in this chapter.

Tip

Keep in mind that low-level DataReader steams are used to return rows from a host of data sourcesnot just SQL Server. If you need to import, query, or correlate data from flat files, Oracle, DB2, Excel, delimited files, or any other data source, you can use other .NET, OleDb, Odbc, or other providers to return a data stream that your application can consume. As I discussed in Chapter 8, you can also pass these DataReader streams to the ADO.NET 2.0 SqlBulkCopy function so the data can be imported into SQL Server.

SqlDataReader Best Practice Designs

Yes, it's possible to read and process each row and column from a SqlDataReader stream one at a time. If you use this approach, you'll also have to hard-code references to each individual column you wish to fetch. Of course, all of this code can be replaced with a single DataTable Load method call that populates a DataTable from the SqlDataReader stream. The downside to using the Load method is that you can't charge as much for your applicationassuming you're charging by the line. Is it faster to fetch rows and columns one by one? Yes, it can be if you don't plan to perform all of the work done by the Load (or Fill) methods. If your program is sifting through the rows one by one and performing logic on the data before you fetch the next row, you might also benefit from this approachassuming that it could not have been done on the server without transporting it to the client in the first place. But what happens when the schema changes? In this case, you'll have to recode the routines that fetch the individual columns, recompile, retest, and redeploy. Next, I discuss how to use the SqlDataReader functions and methods to fetch and process resultsetsif you're still interested.


Populating Data Structures

This section explores the techniques and code needed to populate ADO.NET DataSet, DataTable, Visual Studio 2005generated TableAdapter objects, and custom structures with data without using the row-by-row DataReader Read method. Yes, I've talked about (or around) this before, but I want to take some time to explore the mechanisms behind the scenes so you'll have a better idea of why your application is taking so long to get its job done. I start by discussing application design strategies and how to populate rowsets using the DataAdapter Load and Fill methods, the DataTable Load method, and custom-generated TableAdapter methods introduced with Visual Studio 2005. Because these methods are synchronous (even in the latest versions of Visual Studio), you might want to explore how to run them asynchronously using BackgroundWorker threadsI'll show you how later in this chapter.

Efficient Population by Design

Population (or, more precisely, rowset population) is the process of moving rows from the server to the client and into ADO.NET (or your own) data structures. These rows are generated by SQL Server in response to one or more SELECT statements executed as queries. Properly handled rowset population can play a significant role in the overall performance and scalability of any applicationregardless of the platform. Yes, if you limit the rows returned by a query, the length of time it takes to populate the client data structures is not significant; however, if your application returns more than a few dozen rows, the impact of population increases proportionately. If you multiply the overhead caused by rowset population on network bandwidth and server CPU and RAM resources by the number of client requests, you can get a better idea of how poorly managed population can pull down scalability and overall performance.

By correctly populating your client-side data structures, you can ensure your application performs well and permits other applications sharing the same data to survive. While the techniques used to populate rowsets in a Windows Forms/Smart Client application are similar to those used in ASP, in Web Services, or in business objects, the (good) reasons for doing so are different. Keep in mind that in a Windows Forms application, you have client-side state to manageonce the data is fetched (your client-side rowset objects are populated), the data remains in place. In the other architectures (with the possible exception of business objects), your "client" application should be statelessand cannot depend on persisted rowsets. This means you'll want to save these rowsets in Session state or simply (if one can say that) refetch the rowset state from the server.

Regardless of your design, one of the first questions you need to ask before fetching rows is, "Do I really need the rows to be transmitted to my client?" In other words, you need to decide if what you want to do with the rows could be done better, faster, and easier on the server where the rows currently reside. Do you simply want to display the rows to a human and use them in a report, or process the rows and send back changes to the server? In far too many cases, I see applications that bring rows to the client to make (sometimes complex) changes and attempt to update the server with the new data. The rationale for this approach is that some feel that TSQL is not up to the task of making these changes. In actuality, the problem might just be that the developer (that's you) might not have the TSQL skills to make these changes on SQL Server. Don't feel badlots of Visual Studio developers are a few cards light of a full deck when it comes to TSQL coding subtleties. The good news is that it can be possible to perform these changes on the server using a CLR stored procedure. Sure, the CLR code might take a bit longer to run than the TSQL equivalent, but you'll save as much as one or two (expensive) round-trips per row to query, fetch, and replace server-side rows.

The next question you need to ask is, "Does the data get stale over time?" Unlike that block of yellow cheese you opened last month (or was it last fall) and stuck back in the bottom drawer of the refrigerator, data does not turn a soft shade of green if it's been sitting around too longat least, I don't think it does.

IMHO

Generally, fetching lots of data is like buying lots of ice cream on a hot summer dayit's not much good a couple minutes after you get it.

If the data you're fetching is relatively static, fetching it and even persisting it locally can make a lot of sense. For example, the list of valid U.S. states has not changed since 1959 when Hawaii joined the Union, and even if Washington, D.C., or Puerto Rico or Iraq are granted statehood, keeping a valid states list in a client-side data structure makes a lot of sense, as it can help applications launch more quickly and reduce network and server overhead. Okay, where should these (relatively) static lists be persisted? If you say "in a JET database," go to the back of the class. However, if you say, "Why not save them to a SQL Server Everywhere database?", you win a free trip to Disneyworld. No matter how you plan to build your application, the design needs to consider the volatility of the data being referenced. If the data does not change over time, (ideally) it should be stored locally, but if it's subject to periodic change, you need to fetch as little as possible. If the data is highly volatile (like airline seat status or stock values), you should consider server-side cursors that permit you to easily query the current (up-to-the-instant) value of the server-side row. In this case, the client code fetches keys (pointers) to the server-side rows you need to monitor and subsequently fetches the current value in a separate operation. Nope, server-side cursors are not directly implemented in ADO.NET (as they are in ADOc)but they are implemented in SQL Server, as I discuss in Appendix IV, "Creating and Managing Server-Side Cursors." This means that you'll still be able to open SQL Server cursors, but you'll be responsible for every aspect of the operations, including creating the cursor, opening it with a SELECT statement, fetching the first (and each row), positioning the current row pointer, detecting EOF (and BOF), and performing update-in-place action commands.

Rowset population and this section focus on the rowset(s) returned by the resultset packets generated by SQL Server. In ADO.NET, there are several ways to capture rows being returnedsome are as simple as a single method call like Fill or Load that handles all of the resultsets in a single (synchronous) operation, while others (that I discussed earlier in this chapter) require you to manually loop through each resultset to pick off the rowset column values one at a time. You can also Delegate the responsibility to fetch rows to a "complex" bound control like a DataGridView by binding the DataSource to a DataReader. As discussed earlier in this chapter, if your query execute method returns a DataReader (even asynchronously), you'll have to either fetch the rows (and column values) one at a time or use the new Load method to retrieve your dataI discuss this technique later in this section.

Remember, your code must deal with all of the rows that are returned (or queued by SQL Server to be returned). This means that if you ask for 1,000, 10,000, or a bazillion rows, your code will have to accept all of them before being able to use the Connection to ask for anything else. Consider these strategies:

  • Query only for the rows you need now. It's cheaper to go get more if the user or application needs them than have the server queue up and have to discard rowsets that you didn't really need. Work out a paging strategy that fetches N blocks of Y rows based on a PK range or other blocking criteria.

  • Limit the number of rows returned by using a TOP expression or a WHERE clause that focuses the rowset on a specifically limited set of rows.

  • Choose the right Command Execute method. Use ExecuteReader only when you really need a rowset. If you need only the first value from the first row, you might consider ExecuteScalar, but this won't help efficiency much if your query returns a lot of rows. When you need to execute a TSQL command that performs an action (like an UPDATE), use the ExecuteNonQueryit returns a RowsAffected value and ignores any rowsets generated.

  • If you find that you don't need any additional rows from a DataReader, don't just close the DataReaderthis causes ADO.NET to loop through the rowset fetching all remaining rows as it looks for additional resultsets. Instead, execute the Command object's Cancel method. This might (just might) be able to cancel the operation in progress. In some cases (as when the query is complex), this might force SQL Server to roll back the operation, so be judicious in its use.

  • Leverage SQL Server's ability to execute several TSQL statements or stored procedures in the same batch. This saves round-trips and can use server-side logic to determine how many rows (and which rows) are returned (if any). Remember, stored procedures can call stored procedures, which can call stored procedures. This approach, however, means your code needs to be prepared for "N" resultsets, and all of these might contain rowsetsit can be hard to keep straight.

Regardless of how you execute a query, ADO.NET opens a DataReader behind the scenes to return the SQL Servergenerated resultsets to your applicationyes, even the DataAdapter or TableAdapter Fill methods open a DataReader, which requires an open connection to the data source that must remain open (and viable) throughout the rowset population phase. Until all of the rows you've selected are fetched, the server holds (at least) share locks on the rows and possibly database pages, extents, or tables being referenced. Yes, it's possible to use TSQL commands to manage these locks, but that's beyond the scope of this book and can cause other unfortunate side-effects.

Because you're holding locks on the selected rows, it's imperative that you fetch the rows as quickly as possible to permit other applications to access and update the shared rows. This means you can't let a user or other synchronous operation interrupt the population process. Yes, this means you should not try to read rows with one connection and update them with another. If you catch yourself (or one of the programs you review) doing this, consider performing the operation on the server in TSQL or in a CLR procedure.

IMHO

Never let the user delay rowset population while they decide which changes to make. They might just decide to not decide.

How Does Population Work?

This next section discusses how ADO.NET fetches the results of your queries. If you execute a stored procedure, SQL Server can return several resultsets, each of which can contain the elements shown in Table 11.6.

Table 11.6. Resultset Elements and How to Fetch Them

Resultset Element

Fetch With...

Zero to one rowset generated by a SELECT statement

Read, Fill, or Load methods

Zero to N RAISERROR messages

InfoMessage, Exception handler (sev > 11)

Zero to N informational messages

InfoMessage Event

Zero to N PRINT messages

InfoMessage Event

Zero to N Output parameter values

Output direction Parameters(n).Value at EOF

A ROWSAFFECTED value (unless suppressed by SET NOCOUNT ON)

ExecuteNonQuery method argument

A RETURN value int32

Parameters("ReturnValue").Value

Each of these resultset elements must be fetched in one or more specific ways, as shown in the table. I'll discuss each of these later in this section. Note that your code needs to fetch these elements in the order in which they're returned by SQL Serveras shown in the table. Sure, the InfoMessage, RASISERROR, and other exceptions can occur at any time, but they won't be sent until the procedure ends. Of course, an exception can end the procedure early or simply be a warning, depending on the error severity. In addition, you won't be able to fetch the OUTPUT parameters until the rowset has been completely transported to the clientreached "EOF". The same is true for the RETURN integer Parameter value.

The TDS protocol sends rowsets (if any) back first, followed by PRINT and RAISERROR messages, followed at last by the OUTPUT parameters and the RETURN value. This means you won't be able to access the value of a Parameter object whose Direction property is set to Output, InputOutput, or ReturnValue until the DataReader Read returns False or the DataReader is closed (one way or another). The OUTPUT parameters won't be returned at all if the procedure throws an error.

Tip

As I've said earlier, it's far more efficient to return a dozen OUTPUT parameters than a single row of data.

Populating InfoMessage Messages

Since the earliest days of DB-Library, the InfoMessage has been used to return "informational" messages back to the client application. These include messages generated by SQL Server to report changes in the current catalog (USE statements), changes in language settings, and other system-oriented messages. ADO.NET exposes these in the same way as its predecessorsvia an InfoMessage event. When the TSQL code includes a PRINT statement or executes a RAISERROR with a severity less than 11, the InfoMessage event fires as each message arrives. I set up an example application to illustrate this and try to get a feel for when these events were firing. The code calls a stored procedure that intentionally adds a delay between the PRINT statement to see if the InfoMessage event would fire asynchronously instead of transporting the PRINT strings as they occurit's doesn't. SQL Server caches the PRINT messages while executing the TSQL but does not send them until the final packets of the TDS stream are sentalong with the OUTPUT parameters.

The InfoMessage event handler is passed an instance of the SqlInfoMessageEventArgs that contains the source (always "SqlClient.NET Data Provider"), the Message and an instance of the SqlErrorCollection that contains any "warnings" or low-severity (<11) errors that have occurred. More severe errors cause the SqlException exception to fire. However, you can program ADO.NET to re-route user errors in your procedure to the InfoMessage event by setting the Connection FireInfoMessagesEventOnUserErrors property to True (it defaults to False).[7]

[7] See the example "ManagingPopulation" on the DVD.

The code shown in Figure 11.11 illustrates one (simple) approach to an InfoMessage event handler. It doesn't do much besides displaying the messages and errors in a list boxa practice that's discouraged for a user-interface.

Figure 11.11. Capturing PRINT messages and errors in the InfoMessage event handler.

[View full size image]

Populating the OUTPUT and RETURN Value Parameters

Once the procedure has completedwhich means after all of the resultsets have been returned in the TDS stream, the PRINT, OUTPUT, and RETURN Value parameter values are returned. Fetching the OUTPUT and RETURN Value parameters is as simple as referencing them once the rowset population is complete (as shown in Figure 11.12). The trick to using OUTPUT parameters is to create a named Parameter object for each and setting the Direction property to ParameterDirection.Output. To trap the RETURN value, set the named Parameter object Direction property to ParameterDirection.ReturnValue.

Figure 11.12. Configuring the Parameters Collection, running the query, and capturing the Values.

[View full size image]

Populating with the DataTable Load Method

Introduced in ADO.NET 2.0, the DataTable load method constructs a DataTable from a SqlDataReader. This means (for the first time) you can use a lightweight ExecuteReader call to build a data stream and pass this SqlDataReader directly to the Load method. ADO.NET takes it from thereit uses the schema returned in the SqlDataReader and builds a new DataTable (as shown in Figure 11.13 [8]). This table can be added to the DataSet Tables collection as is or modified to suit your needs. The DataTable load method also supports a LoadOption that determines how existing rows in the DataTable are treatedoverlaid, merged, or "upserted". Yes, "upsert" seems to be a new word made up by the ADO.NET teamthey must have been sleepy that day. Basically, the Upsert option tells ADO.NET to write the incoming values to the "current" version of the column but not to disturb the "original" value.

[8] See the example "DataTableDataSetLoadMethod" on the DVD.

Figure 11.13. Using the Load method to populate a DataTable.

[View full size image]

Populating with the DataSet Load Method

Another addition for ADO.NET 2.0 is the DataSet Load method. It populates a set of DataTable objects using an array of DataTable objects passed to the Load method, which fills each DataTable in the array based on the order of the rowsets returned by the SqlDataReader. In many respects, the Load method is similar to the Fillbut in this case, you build and name the DataTable objects beforehand and use a SqlDataReader to populate the DataTable instances. The example shown in Figure 11.12 illustrates how to build a string array containing the table names and use the DataSet Load method to construct the DataTable objects.

When working with multiple resultsets and the DataTable Load method, it's not necessary to call the DataReader NextResult method to step to the next resultsetthat's done automatically, as illustrated in Figure 11.14. This code runs a query that returns three rowsets and places the rowsets in three new DataTable objects created on-the-fly for this purpose.

Figure 11.14. Stepping through a multiple-resultset query.

[View full size image]

Populating Additional Resultsets

Unless you're using the DataReader, handling multiple resultsets is pretty easythe DataAdapter Fill method automatically builds a DataTable for each of the rowsets returnednot one per resultset (as not all resultsets contain rowsets). On the other hand, the TableAdapter Fill is clueless when you specify a query or call a stored procedure that returns multiple resultsetsafter all, it's a TABLE adapter designed to expose a (single) strongly typed class that maps to a specific rowset.

With the 2.0 Framework, even using the DataReader is not as big an issue as beforeas long as you use the DataSet load method to generate the DataTable objects. If you find that you're in a situation where you must handle the DataReader manually, you'll need to invoke the NextResult method to step to the next resultset.

Using the Fill Method to Populate DataTable Objects

The SqlDataAdapter Fill method executes the SelectCommand and constructs a SqlDataReader behind the scenes. This same approach is used when using a strongly typed TableAdapter and the Visual Studiogenerated Fill method. The GetTable method generated by the TableAdapter works in about the same way.

Yes, the Fill and GetTable methods also handle the SqlConnection, automatically opening and closing the connection as needed. The SqlDataReader data stream is used to determine the inbound rowset schema and to construct the Columns collection. If the Fill method is passed a new (unpopulated) DataSet, ADO.NET constructs and populates a new DataTable object for each rowset returned by the SqlDataReader created by the SqlDataAdapter SelectCommand. If you don't specifically set up the SqlDataAdapter TableMappings collection, the first newly constructed DataTable is named "Table"subsequent DataTable objects are named "Table1", "Table2", and so forth. Yes, you can change this "base" name from "Table" to anything that makes sense by passing the name as an argument to the Fill method, but you'll probably want to "map" the database table names to these DataSet DataTable names configuring the SqlDataAdapter TableMappings collection. Using the DataTableMapping object's ColumnMappings collection, you can also map (and rename) inbound rowset columns to DataTable columns. I illustrate how to execute the Fill method in code in Figure 11.15 [9]. Note that the SQL executed by the Fill method (the SqlDataAdapter SelectCommand) includes two SELECT statements and that each includes the primary key column(s). I name the "base" table "MyTableName", so "MyTableName" and "MyTableName1" are createdwhich is not any more useful than "Table" and "Table1".

[9] See "UsingFillToCreateDataTables" on the DVD.

By default, only "column" metadata is added to the new DataTable objects created by the Fill method. This means no attempt is made to identify primary key (PK) column(s) or set any of the DataTable constraint properties. However, if you set the MissingSchemaAction property to AddWithKey, as shown in Figure 11.15, ADO.NET sets these attributesbut only if the PK columns are included in the SELECT. In this case, the Au_ID and ISBN are the PK columns for the Authors and Titles tables, respectively. In addition, if your query is the product of an OUTER JOIN, ADO.NET does not try to set the PKif you need one, your code needs to set the DataTable PrimaryKey property.

Figure 11.15. Using the Fill method to construct new DataTable objectsone for each rowset.

[View full size image]

If, on the other hand, a populated DataSet or DataTable is passed to the Fill method, ADO.NET has to make several important decisionsand it needs your help. First, consider that using Fill against populated DataSet or DataTable objects implies that you want to either merge the data or append it. If you don't set up a PK or ask ADO.NET to do so, ADO.NET has no choicenew data is simply appended to the end of the existing data. If a PK is set up, ADO.NET assumes that you want to "merge" the data. Inbound rows are matched up based on the PK and the current data is updated. New rows are appended to the end.

Since the inbound schema might not match the DataTable you're populating, ADO.NET has to add any new columns to the DataTable. That's one of the strengths of the DataTableit can be programmatically expanded (or contracted) at any time. The data values for rows that already existed in the DataTable (before you merged in data with new columns) are set to NULL.

Hard-Coding Your Own DataTable Objects

One code-intensive but appealing approach is to write code to define your DataTable object and append it to an existing DataSet object's Tables collection. Once the DataTable is created, you can also add, change, or remove columnsbefore or after you append it to the Tables collection. Unlike ADOc's Recordset object, the DataTable can be easily modified to fit your current needs. The example[10] shown in Figure 11.16 illustrates the first part of this technique. Here, I construct an array of DataColumn objects that are used to populate the new DataTable Columns collection. This code is made much easier to write because I used an Enumeration to define the table column names and order. I know that specific columns have special properties, so I build a Case statement to set those properties. Notice that the DataTable uses an Identity column whose value is generated automatically by ADO.NET at runtime.

[10] See "CreateTableFromScratch" on the DVD.

Figure 11.16. Generate an array of DataColumn objects to populate the DataTable Columns collection.

[View full size image]

Once the array of DataColumn objects is built, your code populates the Columns collection of a new DataTable object using the Add method, as shown in Figure 11.17. In this case, I use both the AddRange method to add the pre-built array of DataColumn objects created in Figure 11.16 and the Add method, which uses the method arguments to set the name and datatype of the new DataColumn. In the latter case, I would have to visit each DataColumn property individually to make other settings. That's the advantage of using the AddRange methodall of the column definitions can be made in a central location. Once the columns have been defined, I set the initial values for all but the first columnwhich is set automatically, as it's an identity value.

Figure 11.17. Creating a DataTable in codesetting the initial values.

Previous Page
Next Page


Fetching Rows Asynchronously

One of the more challenging aspects of working with ADO.NET is overcoming the shortcomings when it comes to asynchronous operations. ADO.NET still does not support the asynchronous Connection Open method or asynchronous rowset population. Fortunately, the 2.0 Framework does include a few more classes to make this more manageable. To illustrate how this is done, I've included an example (a complete program, actually) that handles a number of tasks asynchronouslyincluding fetching a rowset after the DataReader has returned from an asynchronous query. Remember, the asynchronous BeginExecuteReader simply returns a handle to an open DataReaderit does not return a single row of datathat's up to you. I've shown this example at a dozen conferences. It's designed to help use the MSDN subscription index. (You know, the one that takes an eon to locate anything.) The program is used to illustrate how to set up and execute a BulkCopy operation using the new SqlBulkCopy class included in ADO.NET 2.0. The application uploads the data (from a JET database on the MSDN Index CD) to a local SQL Server database (of your choosing). It then permits you to execute rather complex queries against that data. The search UI is shown in Figure 11.18.

Figure 11.18. The MSDN Search Index Utility.

[View full size image]

Once the user initiates a search, the application executes the query asynchronously, and while waiting, it slowly exposes a "Please wait" dialog that fades into view over a period of 5 seconds or so. If the query does not take a long time to execute, the user might never see the dialogjust the results. When the asynchronous portion of the query is complete, the application switches to phase twofetching the rows (rowset population).

The rowset is fetched within a BackgroundWorker thread (new for the 2.0 Framework). While it's beyond the scope of this book to get into the details of how this is managed (and the numerous pitfalls), suffice it to say that the code needed to create and manage the thread is not for the feint of heartbut it's far better than the alternative as implemented by earlier versions of the Framework. The code example shown in Figure 11.19 illustrates the core functionality.

Figure 11.19. Starting a BackgroundWorker thread to complete rowset population.

[View full size image]

The work executed by the BackgroundWorker thread is executed by the DoWork event. This was constructed for me automatically (in Visual Basic .NET) by declaring the bgWorker variable using the WithEvents option, as shown in Figure 11.20.

Figure 11.20. Declaring the bgWorker variable WithEvents.

[View full size image]

The DoWork event handler (shown in Figure 11.21) looks like the other rowset population code I've used before. Just remember that this code is running on another thread. This means it can't reference any object that was not created by the threadthis includes any of the UI elements (TextBox, ProgressBar, or any other Form control). The BackgroundWorker thread handles this issue by setting up a callback routine of its ownthe ProgressChanged event, which runs on the UI Form's thread. It's invoked on demand by the ReportProgress method, which accepts a "percent complete" value (between 0 and 100) to indicate how much work has been done so far. Since I don't really know how many rows can result from the query, I can either fudge a number or go to great lengths to provide a more accurate count. The ProgressChanged event is an opportunity to bump your ProgressBar or otherwise inform the user that the application is still running.

Figure 11.21. The DoWork event handler.

[View full size image]

When the asynchronous load DoWork routine is complete, the BackgroundWorker thread is notified so you'll have an opportunity to tear down and dispose of the thread.

IMHO

This all seems like a lot of trouble to do something that the Framework should have implemented in the first place....


Executing Commands Asynchronously

The ADO.NET proudly rolled out its new asynchronous execute functions with the 2.0 version. Actually, you've been able to execute virtually all ADO.NET operations asynchronously from the beginning: All you had to do was figure out how to set up and execute threadsany average rocket scientist could do it. Of course, these are the same rocket scientists that got meters and feet mixed up with one of their Mars-bound spacecraft. The 2.0 Framework also makes the threading approach easier by exposing the BackgroundWorker thread class. In this section, I'll show you the SqlCommand functions that can be executed asynchronously and how to start, stop, and check to see if they are done. Why bother? Well, if you have users with the patience of Noah, you don't have to worry about providing progress indicators to show that your application is still workingand not locked up (again). For the rest of us, those who have users that start pressing buttons after about 15 seconds while staring at an hourglass cursor, I think you'll find the asynchronous execute functions just what you need.

Do asynchronous functions make sense for ASP.NET or XML Web Services? Sure, when you want to overlap processing parts of the page logic, asynchronous functions make a lot of sense. This way, you can get several queries running, and while they are executing, you can run other code to get ready for the results. Yes, you'll probably end up blocking at the end of the routine waiting for all of the sub-threads to execute, but your page or end-result is built faster than it would be if you executed the queries or performed the other operations serially.

Before I get started, I need to make one thing perfectly clear: Handling threads can be difficult and dangerous. When you start a new thread of execution, you're launching code that executes independently of the code that started the threadwell, almost. Except on multiprocessor systems (or hyper-threading CPUs), a single processor executes all of the threads in the systemyours, Windows, the kernel, and that application you run in the background playing your favorite MPG file. This means all threads are executed serially, but the OS and the processor can interrupt that thread at any time to share CPU time with other threads that need to run. The OS spends considerable time and energy switching context from one thread to another and back again often many millions of times a second.

One of the (many) side effects of the Windows threading architecture is that you won't be able to directly access Windows controls created by another thread. This is a very common mistake. For example, suppose you execute a SqlCommand asynchronously. A new thread is created to do the work, and your current thread (the thread that executed the SqlCommand) continues to run. When ADO.NET signals that the asynchronous thread is done (and your data is ready to retrieve), the second (asynchronous) thread gets the CPU's attention and you're permitted to process the returning rows. What you won't be able to do is set properties on other controls owned by/hosted by the calling threadnot without a Delegate. I'll show you how to use Delegates to get around this problemjust don't go running off into that nearby pasture and set up a threading love-in until I show you where the cows have been.

No, asynchronous operations don't always (not even usually) make sense. They are important for those situations where you don't know how long a command or other operation is going to take place and you need to retain control over your application while this extended process is proceeding. Asynchronous operations can also make sense when you can segment the logic in your routine so operations can be executed in parallel instead of serially. Just use them wisely. Threads, like other system resources, are a precious commodity.

Tip

The MSDN Search sample application found on the book's DVD illustrates use of asynchronous operations.

Coding the Asynchronous SqlCommand Functions

Consider that the asynchronous execute functions call the same synchronous functions behind the scenes, but they do so on their own independent thread, so you don't have to block while you wait for them to complete. When you're informed that the asynchronous query has completed, you'll have an opportunity to fetch data from any rowsets or OUTPUT parameters in exactly the same way, as discussed earlier in this chapter. The fundamental difference is that when executing these functions, you'll have to pass additional arguments to manage and fetch thread status.

The SqlDataReader returned by BeginExecuteReader does only half of the work needed to get your rowset. While it asynchronously executes the query, it signals that it's "done" before a single row is fetched. Unless you pay attention to the later parts of this section, you won't discover how to fetch those rows asynchronouslyI'll show you how to execute the Load (or Fill) function (or any function) asynchronously.

Before you can execute any SqlCommand function asynchronously, you'll need to add another key/value pair to the ConnectionString: ConnectionString Asynchronous Processing =true. This enables the asynchronous features of ADO.NET.

If there is a syntax error deep in the T-SQL code you execute with an asynchronous function, you won't throw the exception in your application until the query completesthis might mean you won't see the error for some time.

Table 11.7. The SqlCommand Synchronous Functions

Execute Function

Returns

Description

BeginExecuteNonQuery

Integer

Executes SqlCommand CommandText and returns rows affected integer

BeginExecuteReader

SqlDataReader

Executes SqlCommand CommandText and returns SqlDataReader (two overloads)

BeginExecuteScalar

Object

Executes SqlCommand CommandText and returns first column from first row from first rowset returned from query or Null

BeginExecuteXmlReader

XmlReader

Executes SqlCommand CommandText and returns System.Xml.XmlReader

Let's walk through a sample application that executes a query that takes a long time to executeI call the stored procedure WaitForALongTime, which includes a WAIT FOR DELAY statement that accepts a parameter to set the delay time. First, I need to build a SqlConnection object that supports asynchronous operations. Figure 11.22 illustrates code to do just this. It also sets up the SqlCommand to call the stored procedure. Note that I set an upper limit of 6 minutes on the SqlCommand, so ADO.NET won't let it run any longer than a quick trip to the fridge for another Diet Pepsi.

Figure 11.22. Set up a SqlConnection that supports asynchronous operations.

[View full size image]

The next block of code sets up the IAsyncResult object, which is used to manage asynchronous operations throughout the Framework. I need to take a closer look at this interface before I move on. IAsyncResult is implemented by any Framework class that can operate asynchronously. Actually, you can call any Framework method or function asynchronously, and I'll have to discuss that later in this chapterfor now, all you need to know is that the IAsyncResult interface exposes several properties that indicate what the thread is doing and has done, and to control its operation. I'll demonstrate their use a bit later in this section.

  • AsyncState: This property returns a user-defined object specified when the asynchronous operation is started. It can be used for anything you need it for. Generally, your code sets or uses this value as it leaves the called asynchronous routine to indicate success, failure, or how its stomach is feelingor anything else it needs to return.

  • AsyncWaitHandle: This property contains a handle created (in this case) by the asynchronous SqlCommand functions. They can be used to wait for the asynchronous operation to complete instead of polling.

  • CompletedSynchronously: This property changes state when the asynchronous operation completes synchronously. I don't recommend using this property to poll the asynchronous operation, as it does not seem to return anything except False.

  • IsCompleted: This property can be used to poll the asynchronous operation addressed by the IAsyncResult interface to see if it's done. Once IsCompleted changes to True, it's safe to tear down any resources used by the thread and to collect the results.

So, we're ready to walk through the first steps of setting up the asynchronous SqlCommand execute function. Figure 11.23 illustrates how this is coded. Notice that I create a new IAsyncResult object and populate it by executing BeginExecuteReader. Your current application thread continues to execute, so you can continue with other work as the new thread executes the CommandText T-SQL. In this case, I'm simply entertaining the user with a progress barthat's what the ShowProgress routine does. Note that I do not simply keep looping as I wait for the asynchronous thread to complete. While that's not a fatal mistake, it would be like starting a race horse out of the gate with a 10-foot rope tied to his tailthe other end of which is tied to your '57 Chevy's bumper. The trick here is to tell Windows that you want it to go off and handle other (more important) operations for a while. In this case, I call the Thread.Sleep function and tell it to let Windows do its thing for 500ms (½ second). Sure, the granularity could be larger, but a user can sense a ½-second difference. A lesser time interrupts Windows more frequently than necessary. Consider that this approach is kinda like that last trip to the lake with your kids. As you drive on, that 4-year-old kid in the back asks, "Daddy, are we there yet?" in a seemingly endless loop (using a pseudo-random number generator to compute the time between queries). After the 119th time, Bill usually says, "Yes! we're here! Just a minute and we'll stop here on the freeway and you can get out and...." That's why his wife goes with him on those trips.

Figure 11.23. Coding the BeginExecuteReader asynchronous function.

[View full size image]

When the IArResult.IsCompleted finally changes to True, it's time to fetch the results. No, the SqlDataReader has not returned a single row at this point. This means you'll need to fetch the SqlDataReader stream using the EndExecuteReader function, which accepts the IAsyncResult object created when you first began the asynchronous operation and returns the SqlDataReader stream.

IMHO

I'm of the opinion that you should never wait forever for anythingexcept a nice piece of carrot cake. So, what if IsCompleted never changes state? A best practice would be to include another counter or watchdog timer in your code to ensure that you eventually exit from the loopyou can't depend on your spouse to nag you.

Waiting for Godot or Your Asynchronous Operation to Complete

There are at least three different strategies you can use to wait for your asynchronous operation to complete. These include:

  • Polling: I already illustrated this approach in Figure 11.23. In this case, you test the IAsyncResult object IsCompleted property until it turns true or you give up waiting. This approach can consume CPU time on your client that might be better spent elsewhere. Just remember not to poll too frequently and spend your time wisely between polls.

  • Callback: This approach is a bit more code-intensive and requires a far more in-depth understanding of how the Framework handles asynchronous operations. Basically, when you start the asynchronous operation, you provide the address of a routine (a Delegate) that is to be called when the operation is done. This approach is not that tough to set up and wastes no time at all while the asynchronous operation is running. I'll illustrate that point next.

  • Waiting: In this approach, you use the ability of the Framework to handle one or several threads at once by synchronizing asynchronous operations using WaitHandle objects. Using another variation of the asynchronous SqlCommand function, you set up a thread-specific WaitHandle that can be passed to the WaitAny or WaitAll primitives to notify you and execute a selected routine when the thread is done. This approach is illustrated later in this section.

Using the Callback Technique to Manage Asynchronous Operations

Since I already illustrated the polling technique, let's move on to callback. This approach requires that you provide a subroutine or function that's invoked (called) when the thread executing the query is done or ready to return the first row. Be carefulthis routine runs on its own thread, independent from the thread that manages your form, so you won't be able to reference any of the controls or other routines owned by the form. This means if you want to display the rowset returned by the query or simply change a component property, you'll throw an exception unless you create a Delegate to do so.

IMHO

Do people think that an application is faster if the progress bar increments more quickly?

This Delegate must be defined with the same signature as the routine you want to call. Okay, if you don't know what "delegates" or "signatures" are, you aren't alone. A Delegate is simply a way to give a name to a subroutine or function declaration and assign it the address of another routine (subroutine or function). This way, when the Delegate is called, it really calls the addressed routinesort of like setting up call forwarding, but without the monthly fee. The Delegate is created with the same number of input and output arguments (its "signature").

Let's say you have a subroutine called "Fred" and you want to call it from another thread. You could declare a Delegate "DelegateFred" and an instance of DelegateFred and point it to the "Fred" function's address, as shown in Figure 11.24.

Figure 11.24. Creating and invoking a Delegate to Fred.

Need a bit more detail? Sure, let's walk through an implementation of the application I just explained, but this time, I'll set up a callback to a routine that extracts the data from the SqlDataReader and builds a DataTable to be bound to the DataGridView control.

First, I set up the SqlCommand Parameter, open the Connection, and create a new AsyncCallback object, which expects the address of the routine to be invoked (called) when the asynchronous SqlCommand execute function is done. I point the AsyncCallback object at the CallMeWhenDone routine. Next, I call the SqlCommand.BeginExecuteReader and pass in the name of our AsyncCallback object. I also pass in a pointer to the SqlCommand object in the AsyncState objectthis is used later in the callback routine. All of this is illustrated in Figure 11.25. I added a line of code to dump the current ThreadID so that I can clearly see which thread is running which code. Just before I end this routine, I enable the Form-based Timer1 control, which increments the ProgressBar so I know that the code is still running. Of course, the fact that the ProgressBar is incrementing doesn't really mean the thread performing the concurrent execution is still running, but it makes my user think so.

Figure 11.25. Setting up an asynchronous execution with an AsyncCallback routine.

[View full size image]

When the asynchronous operation completes, the callback routine is called on its own thread. Now things get tricky. Let's examine the CallMeWhenDone to see how this can be done. First, I need to set up the Delegate object I'll use when it's time to pass the fetched data over to the Form. Remember, I can't access the Form's controls or properties directlythey are running on the other thread. I create a Delegate instance to pass the DataTable created in this thread to a routine that passes the DataTable to the DataGridView.DataSource property.

Once the Delegate objects are set up, I'm ready to fetch the data from the completed BeginExecuteReader routine. This is accomplished by first building a pointer to the original SqlCommand instance that ran the asynchronous query. I use the IAsyncResult.AsyncState argument to pass this address from the calling thread. The EndExecuteReader function returns the SqlDataReader I can use to extract the rows. In typical fashion, I create a new DataTable and populate it with the Load function. When I'm ready to pass this populated DataTable to the Form, I pass it though another Delegate created just for this purpose. Figure 11.26 illustrates how I coded this logic.

Figure 11.26. Capturing the rowset from the EndExecuteReader and passing it to the Form.

[View full size image]

This sample application ("Async Callback" on the DVD) also includes logic to permit the user to cancel the query. I found that this can be problematiceven when you have a separate thread on which to execute the SqlCommand.Cancel function. If ADO.NET senses that the command has progressed too far, it throws an exception. However, if ADO.NET can cancel the operation, the thread executing the SqlCommand throws its own InvalidOperation exception. If you simply step over these exceptions, you should be able to cancel the command. Figure 11.27 illustrates how this is done.

Figure 11.27. Canceling the asynchronous command from another thread.

[View full size image]

Using WaitHandle Logic to Manage Asynchronous Operations

The last of the three techniques you can use to manage threads is a bit more complex in some ways and easier in others. It uses the Framework's ability to manage asynchronous operations by using logic to monitor the state of one or more thread WaitHandles. Since the asynchronous SqlCommand functions return a WaitHandle with the IAsyncResult, it's fairly easy to pick off this handle and simply ask the Framework to block until it's done. Ah, doesn't that mean you're back to blocking after execution? Sure, but since you started the SqlCommand asynchronously, you don't have to block right awaytypically, you run whatever code you can before you need the result of the asynchronous operation. This approach makes the most sense when you start two or more threads working on various asynchronous operations and you need to wait until one or more or all of them are done before proceeding. For example, if you need to launch queries against two or more server data sources at once and have these queries run in parallel, this approach is for you. Sure, you don't necessarily need to be fetching data while other threads are working; you can run several operations of any kind in parallel.

Configuring your code to manage the WaitHandle objects generated by your asynchronous operations is easy. All you need is an array of WaitHandle objectsone for each SqlCommand asynchronous execution function you want to monitor. When you use the Begin asynchronous method, capture the WaitHandle property from the IAsyncResult that's returned by the asynchronous method and store it in the WaitHandle array.

When you're ready to see if your asynchronous operation is done, set up your code to call the WaitHandle WaitAny function and point to the WaitHandle object array you just built. At this point, your code blocks until one (any one) of the asynchronous events signals that it's done or the timeout expires. The WaitAny function returns an index (integer) that indicates which of the WaitHandle objects is done. Once you process the results of that operation, you'll need to come back and continue looking for additional threads to complete.

If you need all of the asynchronous operations to complete before you're notified, use the WaitAll function. In this case, your code is blocked until all of the threads in the WaitHandle array are done or the timeout expires.

When you configure the WaitHandle object, you can (should) set a timeout to tell the Framework how long to wait for the operation to complete. As you're well aware (or should be by now), stuff happens. If the query gets lost or confused, you're going to want a backdoor to let your logic out.

No, you won't be able to start a timer to increment a ProgressBar while the WaitAny or WaitAll functions are executing. Your UI thread is not getting any cyclesthe Framework is attending to other applications and won't give you control until the threads are (all) done or the specified timeout elapses. This also means you won't have any cycles to cancel the querynot after you've executed the WaitAll or WaitAny functions.

MARS introduces a new wrinkle to this picture. Since you can execute several operations at once using MARS, you would think that you can execute several simultaneous queries at once on the same connection. Well, you can'tMARS does not permit multiple asynchronous operations.

That's a lot of information to consume without looking at the code used to implement it. The example I'm going to walk through ("Waiting on Thread Handles" on the DVD) is designed to execute three independent operations on three different connections. This permits the server to run all three queries in parallel (whenever possible) or the approach could be used to execute three (or "N") queries or action commands to three separate and entirely different servers. When the Windows Form is first set up, I create these SqlConnection objects and three SqlCommand objects. The Form_Load event is illustrated in Figure 11.28. Note that I use a special version of the PublishersBy-Author stored procedure that waits for a few seconds before returning, so I can test the affects of the SqlCommand objects returning out of order.

Figure 11.28. The example sets up three separate SqlConnection and SqlCommand objects.

[View full size image]

The BuildAndExecuteAsyncCommands routine is responsible for finalizing the configuration of the SqlParameter objects (they all require an input parameter to indicate which author is desired), opening the SqlConnection objects, and executing the BeginExecute function against all three SqlCommand objects. The first section of this routine subsequently captures each WaitHandle object from the IAsyncResult objects returned by the BeginExecute functions and places them in the WaitHandles array. Figure 11.29 shows how this is done in code.

Figure 11.29. Finalizing configuration of the SqlCommand objects and executing them.

[View full size image]

The next step is up to you. At this point, you're free to execute whatever code needs to be (or can be) completed before waiting for the queries to complete.

When I'm ready to wait for the threads to complete, I set up three SqlDataReader objects to capture the returned rowsets. The example code then uses the WaitAny (in this case) to block the thread and wait for the asynchronous operation to complete (the ones I started with the BeginExecute functions). As they finish, their individual WaitHandle objects (in the WaitHandle array) change state and the WaitAll function returns the Index. At this point, I know that the asynchronous operation completed or it timed out. Notice that I gave the operation 12 seconds (12,000ms) to completethat's a really long time for any query to run. The logic in Figure 11.30 first tests to see if the specified thread (as indicated by the Index (Integer) returned by WaitAny has timed out. If so, I throw an exception and keep waiting for the other threads to complete or timeout.

Figure 11.30. Wait for the threads to complete.

[View full size image]

At long last, I'm ready to capture the rows returned by each of the threads. I use the Index returned from the WaitAny function to choose which SqlCommand to process. This is illustrated in Figure 11.31. This part of the code should look pretty familiar by now.

Figure 11.31. Process the rowsets from each SqlCommand executed.

[View full size image]

As far as the example goes, there's little to do now but clean upclose the connections and free up the threads and other resources I allocated. The DataTable Load method I used in the ShowData routine closed the DataReader for me.

While this example application has a "Cancel" button, it's not much use. I don't waste any time between the time I execute the asynchronous commands and the time I start waiting for the results.


Summary

One of the most important aspects of executing any T-SQL against SQL Server is being prepared for the results. Sure, this means capturing the rowsets that return as well as the OUTPUT and RETURN parameter values, but it also means trapping the exceptions. The topics I visited in this chapter should give you a better understanding of how to execute a SqlCommand either synchronously or asynchronously and deal with the resultswhether or not the query did what was intended. Using these techniques, you can pull down rowsets and manage complex resultsets and deal with the events and exceptions these operations can invoke. I also talked about how to manage the low-level data stream as well as how to process the rowsets and returned parameters. Since stored procedures and the OUTPUT parameters are an integral part of many designs, I'm sure you'll find these techniques useful. Just be prepared for the stuff that happens when you execute any T-SQLyou never know when SQL Server will get fussy and talk back.


Chapter 12. Managing Updates

    Introduction

    Using Wizards to Generate Update Action Commands

    Update Alternatives: Using the TableAdapter Configuration Wizard

    Complex Updates with Server-Side Logic

    Managing Concurrency by Design or Collision

    Summary


Introduction

Most of the folks who create data access applications eventually need to save data back to the database. There are any number of strategies that you can use to accomplish this using Visual Studiosome of which involve drag-and-drop update or wizard-generated code that Visual Studio cobbles together based on the SELECT statement. Other, somewhat more complex approaches require that you create the T-SQL action queries yourself or depend on existing stored procedures to make the changes. Sure, Visual Studio has gotten (a bit) smarter when it comes to generating UPDATE statements at design time, but it's still fairly limitedas you'll discover later in our tour of this topic.

When it comes to updates, the fundamental problem with Visual Studio is that it takes a relatively simplistic approach. It assumes that your database is simply a collection of individual tables that can be updated willy-nilly. Most serious databases don't fall in this categorythey are relational in every sense. They include many parent-child/primary/foreign-key relationships and don't take kindly to changes that don't follow a fairly rigid discipline. For example, you must add parent rows before child rows and delete children before deleting their parents. At this point in time, the Visual Studio code generators do not deal with this reality.

When I poll a conference session filled with Visual Studio/SQL Server developers, 90% of them don't expose the base tables at allthey expose stored procedures or views to protect the data and referential integrity, whether or not they're using SQL Server. I've talked about this on any number of occasions here and in my sessions. These stored procedures are usually fairly sophisticated, as they accept parameters that are fed to not one, but perhaps a half-dozen or more UPDATE statements in a single atomic operation. Visual Studio does little to help with this (fairly common) paradigm, so until they catch up, we (you and I) are on our own. This chapter should help you understand the limited amount of help Visual Studio can provide, as well as get a handle on what you'll have to do to fill in where Visual Studio exposes gaps in the roadway. I'll also show you how to set up calls that bypass the DataAdapter or TableAdapter Update methodsa technique you'll need when your database gets more sophisticated.


Using Wizards to Generate Update Action Commands

Visual Studio includes a number of wizards, including the (now obsolete) DataAdapter Configuration wizard (DACW) and its replacement, the new Data Source wizard. It seems that Microsoft wants developers to use its new TableAdapter over the DataAdapter for reasons I discussed in Chapter 6, "Building Data Sources, DataSets, and TableAdapters." The TableAdapter is generated by the Data Source wizardit's used to provide much of the functionality of the DataAdapter but provide the functionality exposed on the DataSet to the DataTable.

IMHO

I'm of the opinion that the DACW has been around too long to be unceremoniously shoved into a ditch at the edge of town like a derelict Mercedes.

The problem with the DACW and several other code generators used by Visual Studio is their heavy dependency on the SqlCommandBuilder. Sure, there are times when you (and I) find it's just simpler to use these wizards to generate ADO.NET code, and while we wince, we still build demos, prototypes, and "quick and dirty" applications that use these tools.

IMHO

In Chapter 6, we toured the SqlCommandBuilder when I discussed building updateable DataSets and DataTables. You should already know why I don't recommend its use.

What the CommandBuilder Is Supposed to Do

When you have a simple table-based query that only references a single table and you want to be able to update the table's rows, someone has to build a set of action commands to do the job. That's what the CommandBuilder is supposed to do. This relieves you of the responsibility of writing your own action commands (in these simple cases) by automatically constructing the T-SQL action queries, ADO.NET Command objects, and their associated Parameters collections given a valid SelectCommand. Cool. And it works, but only in a very limited and relatively inefficient way, and for a price (as I describe in this chapter). However, unlike DACW-generated code, the commands created by the CommandBuilder are unseenin another black box (like the notorious Visual Basic 6.0 Data Environment Designer).

Tip

One can use the CommandBuilder indirectly (as when using a Visual Studio data access code-generating wizard) or directly in your own code, as when you have not built a strongly typed TableAdapter and need an updateable DataSet. Frankly, neither one of these techniques is very appealing.

When the CommandBuilder is called by the TableAdapter or DataAdapter wizards at design time, Visual Studio extracts the generated code from the CommandBuilder and uses it to build strongly typed classes that reflect the schema of the data referenced in a provided SELECT statement. The CommandBuilder assumes (requires) that the SELECT statement is relatively simple. If it contains a JOIN, refers to a stored procedure, or generates multiple resultsets, the CommandBuilder can give up as quickly as my granddaughter when asked to eat broccoli. I'll dig into these details later in this chapter.

One other important point: The CommandBuilder does not generate any post-action fetch statements. This means that you (or Visual Studio) must add fetch queries to the INSERT and UPDATE commands to retrieve the current values in the changed row. For example, once a row is added, you'll need to fetch the new identity value. The same is true for the new TimeStamp value on any row added or changed. Fortunately, Visual Studio does this for you by adding the needed fetch queries to the generated UpdateCommand and InsertCommand. ADO.NET is there ready to help post these updated column values back to the in-memory DataTableassuming it's correctly programmed.

Generating the Action Commands

In order to create an updateable DataTable in ADO.NET, someone or something (in this case) has to generate three Command objectswell, four really. The first (the SelectCommand) is used to fetch the initial rowset using a (very simple) SELECT statement. The other three Command objects are (eventually) built by the CommandBuilder using this SELECT statement, so it's an important first step. These four Command objects are named as follows:

  • SelectCommand: Specifies the SQL code and parameters needed to fetch the rowset(s) based on a single (simple) SELECT query. Yes, you can specify more complex SELECT queries and even include a JOIN, but if you do, you'll find that the CommandBuilder can't generate the remaining Commands.

  • InsertCommand: Specifies the SQL code and parameters needed to insert a new row in a database table.

  • DeleteCommand: Specifies the SQL code and parameters needed to delete a specific row from a database table.

  • UpdateCommand: Specifies the SQL code and parameters needed to change a specific row in a database table.

The last three of these Command objects are created for you automatically and managed behind the scenes by the CommandBuilder. These so-called "action" commands (the InsertCommand, DeleteCommand, and UpdateCommand) are generated by the CommandBuilder class based on the columns and WHERE clause specified in the SelectCommand's SELECT statement. The CommandBuilder also requires that a query be executed against SQL Server to return metadata that describes the SELECT. This costs at least one full round-trip to the server to accomplish, so it can potentially hurt performance fairly badly if not handled wisely.

Consider that ADO.NET developers are responsible for supplying all of the SQL code needed to fetch and change the data in the target database tablethere are no "black boxes" that congers T-SQL on-the-fly in ADO.NET (like there is ADOc). This means that unless you write the code yourself, you'll have to depend on classes like the CommandBuilder, Visual Studio (which calls the CommandBuilder), or other third-party tools to generate it for you.

Handling ConcurrencyUpdate and Insert Collisions

The concurrency problem that you and the data access interfaces have to deal with is that you (should) want to know if the data row in the database table you're changing still (doesn't, or already) exists, and if the data has changed since you initially fetched the data. This is called "concurrency management." It's a concept you have to be concerned about even in a single-user database. In multi-user databases that are designed to permit more than one client access to the same row(s), the task of managing concurrency gets far more complex.

IMHO

I'm of the opinion that it's better to build concurrency into an application's fundamental design instead of tacking it on later. It's kinda like designing city streets and highways to include overpasses and traffic lights that prevent cars from running into each other instead of figuring out where to position ambulances and tow trucks to cart off the bodies and crashed cars after the collisions.

Let's go back in time for a moment to ADOc. When you use the Recordset Update method, ADOc generates the SQL to make the changes to the database at runtime. This means it creates appropriate INSERT, UPDATE, and DELETE statements on-the-fly to post changes made to the rows in the Recordset.

In the process of generating the action commands on-the-fly, ADOc uses the Update Criteria property, along with extra metadata, fetched (that bulks up the Recordset) to adapt the type of concurrency code it generates to match your specific needs. For example, you can choose from four different types of concurrency management in ADOc by using these Update Criteria settings:

  1. adCriteriaKey: Find the row to update using only the primary key. This (essentially) was a "last wins" update strategy.

  2. adCriteriaAllCols: Compare all columns of the current row's original values with the existing row's values. If they match, ADO thinks that no other changes had been made since the row was last read. Of course, the row could have been changed a million times, but as long as the values were set back to the original state, this concurrency option would not notice. Note that this "shotgun" approach is the only option available in the SqlCommandBuilder.

  3. adCriteriaUpdCols: Compares only those columns that have changed with those currently stored in the database. If these match, the UPDATE succeeds.

  4. adCriteriaTimeStamp: Compares the row's TimeStamp column with the value in the database. If the TimeStamp columns are the same, the row has not been modified and the UPDATE succeeds.

The problem with this runtime T-SQL-generation approach is that it's also not particularly flexibleyou have no control over the way the code is generated beyond the four Update Criteria settings. You still can't tune the T-SQL to meet special-case needs. As you'll see with ADO.NET, ADOc also makes some pretty challenged decisions when it comes to handling rights (permissions to update specific columns).

How Does ADO.NET's CommandBuilder Handle Concurrency?

ADO.NET 1.0 was lambasted when it first came out because the "new and improved" CommandBuilder implementations (for all .NET Data Providers) use a "shotgun" approach to generate Command object codeit tests for all columns in the target row against the original row fetched by the SELECT, as shown in Figure 12.1. Notice that this T-SQL UPDATE statement (automatically generated by the CommandBuilder) tests each column's original value against the current server-side value. If all match, the UPDATE succeedsas long as the PK value is found.

Figure 12.1. The default CommandBuilder-generated UPDATE statement.

This approach is especially challenged in that it ignores that some columns you SELECT might be read-only and not meant to be included in the UPDATE statement. For example, those columns you don't have sufficient rights to change, like Salary, are included in the UPDATE.

To address some of these issues, the Microsoft team added a new ConflictOption to the CommandBuilder to help expose some of the ADOc functionalitythis new property goes partway toward a solution. Let's take a quick look at this property to see how it can be used to help manage (simple single-table) concurrency problems. These new option switches are exposed in the Data Source wizard, as discussed in Chapter 6.

The ConflictOption property can be set to any of three options:

  1. CompareAllSearchableValues: When set (the default), UPDATE and DELETE statements are generated to include T-SQL that compares all selected column original values with the same columns in the server table. This does not include expressions returned by the SELECT or columns you add to the DataTable manually in code. The code in Figure 12.1 was generated using this option.

  2. CompareRowVersion: If a Timestamp column is present, it's used in the WHERE clause to determine if the row has been changed since the data was fetched. If the value is unchanged, the UPDATE or DELETE proceeds.

  3. OverwriteChanges: In this case, the generated UPDATE code merely verifies that the target row for an UPDATE or DELETE exists by matching the PKif the row exists, the action is executed.

Given that the ADOc approach required quite a bit of help from the fetched metadata (which made the Recordset much larger than it had to be) and didn't work particularly well with some providers, Microsoft found it easier to take a bare-bones approach with ADO.NET. This is understandable because many developers didn't expect ADO to construct the action commands anywaythey had already moved their action command logic to stored procedures. Having ADO.NET do more than absolutely necessary was also contrary to its charter: to be light, fast, and easy. It was also important to simplify the construction of .NET data providers so it would not take an OLE DB brain surgeon to build them. This strategy plopped the responsibility for the action query SQL back on the ADO.NET developer. Unfortunately, that's kinda like dropping an angry porcupine in your date's lap.

How the ADO.NET CommandBuilder Works

The CommandBuilder[1] expects you to provide a viable, executable, but simple SelectCommand associated with a DataAdapter. It also expects a viable Connection. That's because the CommandBuilder opens the Connection associated with the DataAdapter and makes a round-trip to the server each and every time it's asked to construct the action queries. It closes the Connection when it's done. This round-trip query can be reproduced in your application if you want to get a better idea about what's being done behind the scenes to determine the table schema. It's also a good tutorial on the DataReader class's GetSchemaTable. Figure 12.2 illustrates the code to fetch the table schema from the Authors table.

[1] We use the term "CommandBuilder" here as the discussion applies to all implementations of the CommandBuilderincluding the SqlCommandBuilder implemented in the SqlClient namespace.

Figure 12.2. Gathering schema information using the DataReader class.

[View full size image]

In this example, our code does what the CommandBuilder does behind the scenesit returns a DataTable containing the schema details for the Authors table. We used SELECT * to get the query to return an in-depth schema report that includes all columns (as shown in Figure 12.3). Note that the Au_ID column has "IsKey" checked. This means that it's the table's primary key (or it's one of the columns that constitutes the PK). Once this DataTable is returned, the values it contains are used to generate the action queries. So, if this query does not indicate that there is a PK, the CommandBuilder won't be able to generate any of the action queries.

Figure 12.3. DataTable created by GetSchemaTable.

[View full size image]

No, ADO.NET does not use the new 2.0 GetSchema method to uncover the query schema as we discussed in the last chapterperhaps because it did not exist years ago when this approach was originally designed.

Sure, GetSchemaTable works even when the query includes a JOIN or other complex logic, and it can be executed against a stored procedure or a view. That said, the CommandBuilder requires a pretty simple query that addresses a single database table. When you use the CommandBuilder against a query that returns duplicate columns or multiple tables, the CommandBuilder parser gives up trying to figure out how to build the action queries. Remember, the DataAdapter Update method is designed to provide updateability for a specific table. And behind the scenes, the DataAdapter and Data Source wizards call the CommandBuilder to generate some of their code.

Tip

Sure, it's possible to create your own CommandBuilder or custom wizard to generate the action queries you need. We suspect there are already several of these in the marketplace.

Coding the CommandBuilder

The code to add a SqlCommandBuilder to your application is shown in Figure 12.4.

Figure 12.4. Creating a SqlCommandBuilder in code.

[View full size image]

What you can't see from this code is what's going on behind the scenes. However, even if you turn on the SQL Server Profiler (a really good idea when working with Visual Studio code generators), you still can't tell that construction of the CommandBuilder does anything, at least, not until the code executes the CommandBuilder GetUpdateCommand method. That's because the CommandBuilder class does not construct the action commands until they are first touchedas they are needed by the application. When we execute the GetUpdateCommand method that generated the SQL for the UpdateCommand (as shown in Figure 12.4), ADO.NET executes the T-SQL shown in Figure 12.5.

Figure 12.5. T-SQL used to generate the action queries based on the SELECT statement.

[View full size image]

This code (generated by the GetSchemaTable method) returns a resultset that includes just the metadata for the table in the SELECT statement (FMTONLY=True). This resultset is used to construct the three action queries (InsertCommand, UpdateCommand, and DeleteCommand) associated with the DataAdaptereven though the same query was used to populate the DataAdapter's DataTable.

Remember, unless the SelectCommand is something more than a simple SELECT, as we mentioned earlier, the CommandBuilder cannot generate the action commands. It also gives up if the SelectCommand is too complex or contains a JOIN. There are several other limiting factors as well:

  • Your SelectCommand must retrieve at least one primary key or unique column as part of the query. Just having a TimeStamp column without a primary key of some kind won't work.

  • Your SELECT statement must refer to a single table; thus, the SelectCommand cannot refer to SQL queries, stored procedures, or views that contain JOIN operators.

  • The SelectCommand must refer to columns that permit read/write operations. You can't include columns or expressions that can't be written to for some reason.

  • The CommandBuilder makes no attempt to (nor does it provide any mechanism to) fetch output arguments from the action query (including the Identity value).

  • If the CommandText, Connection, CommandTimeout, or Transaction properties of the SelectCommand change, you have to notify the CommandBuilder by executing the RefreshSchema method, which takes another round-trip.

  • The CommandBuilder-generated UpdateCommand or DeleteCommand does not change any row that's been modified since last read. This means you'll have to write your own action query to "force through" an UPDATE or DELETE operation if that's how your concurrency scheme is designed.

  • The CommandBuilder is designed to work with single, unrelated tables. That is, you can't expect to update tables with primary key/foreign key relationships.

  • If the SelectCommand SQL columns contain special characters, such as spaces, periods, quotes, or non-alphanumeric characters, you can't use the CommandBuilder.

When the .NET Framework Version 1.1 shipped (along with Visual Studio .NET 2003), CommandBuilder support was added to both the OracleClient and Odbc .NET Framework data providers. No, the CommandBuilder is not universally supported by all .NET Framework data providers, so don't be surprised when your provider balks when you try to use it.


Update Alternatives: Using the TableAdapter Configuration Wizard

Okay, so it's tough to use the CommandBuilder (and, thus, the DACW or the TableAdapter) when your SELECT queries get a little complex. What other choices do you have when you need to generate and execute the action commands? It really depends on your overall update strategy.

Updating with the TableAdapter

If you want to use bound controls and let the DataAdapter or TableAdapter Update method handle the changes, consider that the DataAdapter and TableAdapter are table-based, not set-based, paradigms. They're designed to work with a rowset "bound" to a specific (single) table, so they might not really be suitable at all. Sure, it's possible to simply reference a single table with a (fairly) simple SELECT statement and let the TableAdapter configuration wizard fill in the appropriate T-SQL code to update the database. Yes, this code is generated by the CommandBuilder at design time and supplemented by Visual Studio to return changed data from the new and updated rows.

If the Data Source fetch query steps over the line as far as the CommandBuilder is concerned, the TableAdapter drag-and-drop paradigm leaves the "Save" icon disabled as shown in Figure 12.6, and the "Create methods to send updates..." option disabled in the TableAdapter configuration wizard (TACW), as shown in Figure 12.7.

Figure 12.6. The Visual Studio drag-and-drop code generator does not always enable update logic.

[View full size image]

Figure 12.7. The TACW won't let you choose to automatically generate update logic in some cases.

[View full size image]

In this case, the TACW references a simple parameter-based stored procedure to fetch the initial rowset. The StateWanted and MaxRows input parameters are handled in the UI, but the "Save" icon is grayed out.

If you insist on using the Update command against several related tables, you'll have to write custom code to handle each table separately and figure out how to override the default behavior by trapping the OnRowUpdated, RowUpdating, and/or RowUpdated events and fire your own procedures to perform the changes as described later in this chapter.

Updating Tables with the TableAdapter

Yes, if your SELECT statement is simple, Visual Studio and the TACW can generate all of the action commands. You can even specify how concurrency is managedleveraging the new UpdateOption property. To access this option:

1.
Create a TableAdapter that specifies a simple SELECT in the fetch query. In this example, I simply checked the "Customers" table in the list of available tables, views, and stored procedures. This generates a SELECT query that includes all columns from the table without a WHERE clause. Of course, this also assumes that you plan to choose a mechanism to limit the number of input rows. I'll revisit that issue in a minute.

2.
Open the TableAdapter in the Data Sources window by right-clicking on the DataSet and choosing "Edit DataSet with Designer".

3.
Right-click on the DataSet designer dialog and choose "Configure". This opens a dialog that permits you to alter the SELECT statement generated by the TACW and set the Advanced options, as shown in Figure 12.8.

Figure 12.8. Setting advanced concurrency options for a simple SELECT query.

[View full size image]

4.
The Advanced Options dialog (shown in Figure 12.9) permits you to change how the TACW generates the action queries and other advanced options.

Figure 12.9. The TACW Advanced Options dialog.

As you can see, the Advanced Options dialog does not expose all of the UpdateOption property settings. It does automatically sense if there is a TimeStamp column in the fetch query, and if there is, the TACW uses this column to manage concurrency. You can also choose to disable the post-change fetch, which returns the newly added or updated row to keep the local TableAdapter object's rowset in sync with the database row.

Using Stored Procedures for Action Commands

So, how can you specify your own stored procedures to fetch and update the data using the TableAdapter configuration wizard? That's not too hard; just follow along.

1.
Create a TableAdapter and specify the appropriate stored procedure to fetch the rows. Note that the TACW will help manage any input parameters for you when you use drag-and-drop to populate UI elements on the form. This is done by choosing a stored procedure from the list of available objects (Tables, Views, Stored Procedures) when initially setting up the TableAdapter.

2.
Open the Data Sources window in Visual Studio and right-click on the selected Data Source TableAdapterchoose "Configure DataSet with Wizard."

3.
Choose the appropriate stored procedure from the list of visible stored procedures, as shown in Figure 12.10, and click "Finish."

Figure 12.10. Select the rowset-fetching stored procedure.

[View full size image]

4.
You're now ready to specify the action command stored procedures. Of course, these procedures depend on the columns returned by the initial fetch stored procedure, so you'll need to make sure that the correct columns are returned. Right-click on the TableAdapter again and choose "Edit DataSet with Designer".

5.
In the TableAdapter designer, right-click on the TableAdapter and choose "Configure". This opens a TableAdapter Configuration Wizard dialog, shown in Figure 12.11. Using the drop-down menus, choose appropriate stored procedures for each action command to implement. Click "Next" or "Finish" to commit these changes.

Figure 12.11. Choose appropriate stored procedures for each action command.

[View full size image]

6.
At this point, the TACW generates the TableAdapter code to configure and execute the selected action commands.

Now, when your application calls the TableAdapter Update method, the appropriate stored procedure is called. This includes populating the stored procedure parameters collection, invoking the stored procedure, performing a post-action query to update client-side values, and posting these changes to any bound controls.

Managing Generated Parameters

So, if you set up the DataAdapter or TableAdapter with the Visual Studio wizards, quite a bit of code is generated for you to handle several housekeeping functions. For example, I talked about how it's important to fetch the newly created identity PK or TimeStamp value post-insert or update operations. ADO.NET is designed to be prepared for these values either in a row (per action command) or via OUTPUT parameters. To get ADO.NET to automatically post these values back to the client-side rows, your code must bind Parameter objects to the source columns. As I said, the wizards automatically generate this code for you, as shown in Figure 12.12. Ah, this code has been modified to fit on the pageit was all on one (long) line before I added the space underscore(_) breaks. Note that the Parameter definition includes pointers to the SourceColumn and SourceVersion. This permits ADO.NET to postback the new TimeStamp (in this case) to the source table after the InsertCommand has executed.

Figure 12.12. TableAdapter configuration wizardgenerated code to post back the new TimeStamp value.

[View full size image]

This code is found in the project, buried under the generated DataSet as a .vb file. I suggest you dig into this file to see what the wizard generated for you. There will come a time when you'll want to clone some of this functionality yourselfjust use this generated code as a template. A few notes here:

  • Note that the wizards don't try to make this easy to readit's designed to run quickly. Your code could use a With statement to make it far shorter and more succinct.

  • The code also spells out each of the object references. Your Imports statement can deal with object addressing in far fewer words.

  • The Parameters are all defined as InputOutput because of an (ancient) bug in the interfaces that prevents ADO from determining the real direction for OUTPUT parameters. There is nothing to stop you from setting the correct direction.

IMHO

No, it's not a good idea to change any generated source code.

Configuring Parameters with Visual Studio

No, it's not a good idea to manually change generated source code. If you do, you'll invariably find your changes have been replacedVisual Studio regenerates code at every opportunity and late at night while you're sleeping (or it seems so). Yes, you can safely extend the generated code (partial) class with your own class code, but that's beyond where I want to go in this book. I also expect Microsoft to change how these partial classes are managed in the next version, so I'll hold off writing about it until then.

It is a good idea to use the copious IDE wizards and property pages to modify the attributes used to generate the TableAdapter and DataAdapter code. Consider that the TACW (and DACW) simply gather these attributes from you and the database schema to generate the code. You can choose to start over and rebuild the TableAdapter or DataAdapter from scratch, but you might find it easier to simply tune the attributes using the TableAdapter Properties dialog. In this case, I've drilled into the same TableAdapter whose generated code is shown in Figure 12.13.

Figure 12.13. The TableAdapter Properties dialog.

Getting to this dialog is really pretty easyopen the .xsd DataSet in "designer" mode, right-click on the TableAdapter (CustomersTable, in this case), and choose "Properties." As you can see, this Properties dialog permits you to edit every aspect of the generated code, including the fetch query and the action queries used to fetch and change the underlying DataTable. You can use this property page to change what the wizards generate and go well beyond the options provided in the wizard dialogs. For example, if you want to use an ad hoc for the SelectCommand and InsertCommand but a stored procedure for the other action commands, you can configure that from here.

When you drill into the Parameters (Collection) item in the Properties dialog, you're presented with another dialog to fine-tune the individual Parameter object property settings (as shown in Figure 12.14).

Figure 12.14. Configuring Parameter properties.

[View full size image]

Notice how the Parameters Collection Editor dialog permits you to set the @NewTimeStamp properties dialog directly. Again, note that this column is "bound" back to the source table. So, when ADO.NET executes the InsertCommand (via the Update method), the newly assigned TimeStamp value is applied back to the source table.


Complex Updates with Server-Side Logic

When working with more complex relational data structures, especially where you must update more than just a simple parent-child hierarchy, a common approach is to build and execute a set of stored procedures that handle the action command operations so the transaction, update, and collision-detection logic is all handled on the servermuch as we just did using the TableAdapter Configuration wizard. Frankly, this is how updates are handled in many applications, as it is least dependant on the vagarities of the data access interfaces.

In these situations, you can build a set of Command objects to capture the parameters required for the underlying action commands implemented as stored proceduresbasically, mirroring what was done by the TACW in the previous examples. Yes, this is a lot of trouble and requires that you build some fairly brittle code. This means if the stored procedure signatures change, you'll find yourself back modifying the code to adapt the stored procedure changes to your application. Of course, the TableAdapter approach is just as brittlebut in that case, you only have to figure out how to refresh the code generated by the TableAdapter Configuration wizardnot a simple feat, as I discussed in Chapter 6.

Using Events to Execute Your Own Update Logic

If you plan to use the SqlDataAdapter Update method, you'll have an opportunity to call your own server-side executable code or simply execute your own action commands by trapping the RowUpdating event. This is not an option when working with the TableAdapter unless you add these events yourself. When you're working with UI controls bound to a DataTable, the DataAdapter Update method can be very attractive. Sure, it does not do much, so it would not be that hard to replicate its functionality, but this is code you have to write, debug, and train your organization to use. In this case, it's your responsibility to handle every aspect of the update process, including:

  • Populating input and input/output parameter values

  • Executing the appropriate Command (s) based on the DataRow RowState property

  • Capturing the resultset RowsAffected value and updated rowsets from the action commands

  • Testing RowsAffected for an acceptable value

  • Posting changes to the underlying DataTable rows

  • Resetting each DataRow object's RowState property to reflect the change, assuming it worked, or call AcceptChanges once all rows have been processed

  • Handling concurrency and other exceptions

  • Repeating, as necessary, for all DataTable changes

Tip

The TableAdapter does not expose the RowUpdating or RowUpdated events unless you code them yourself.

Once you execute the Update method, calling one or more stored procedures instead of (or in addition to) the DataAdapter action commands is done by "hooking" the DataAdapter Update method events. When the SqlDataAdapter Update method is executed, two events are fired as each data row is updatedunless you're in "batch" mode, which I discuss in a minute.

The sequence of operations is as follows:

  1. The values in the DataRow to be updated are moved to the parameter values passed by the Command Parameters collection.

  2. The RowUpdating event is raised. Here, you have an opportunity to execute your custom codeeven other update operationsbefore the action command is passed to the server for execution. You can also prevent the action command from being executed. Arguments passed to this event include[2]

    [2] These arguments are different for batch modeI'll get to that in the next section.

    • Command: The Command to be executed.

    • Errors: Errors generated by the provider when the command executes. Since the Command has not executed yet, it's not clear what errors would appear in the RowUpdating event. This makes more sense in the RowUpdated event.

    • Row: The DataRow to be updated.

    • StatementType: The type of statement to be executed (update, insert, delete, batch, or select).

    • Status: Determines the action to take when exiting the OnRowUpdating event handler. These include:

      • Continue: The DataAdapter is to execute the CommandText.

      • ErrorsOccurred: Tell ADO.NET that the Update operation failed.

      • SkipAllRemainingRows: Tell ADO.NET that no further rows should be processed by the Update method.

      • SkipCurrentRow: Tell ADO.NET that the current row is not to be processed by the Update method.

  3. If the action argument of the RowUpdating event handler is set to Continue, the CommandText is passed to SQL Server for execution.

  4. If the Command is set to FirstReturnedRecord (a DataReader execution option), the first returned rowset is placed in the DataRow.

  5. If OUTPUT parameters are returned by the query, they are placed in the DataRow (in the Parameters collection).

  6. The RowUpdated event is raised. Here, you have another opportunity to execute other codeeven other update options. The event arguments include the newly updated row and the number of rows affected by the CommandText action command.

  7. AcceptChanges is called. This resets the RowState of each row in the DataTable to "Unchanged" to indicate that all changes have been made.

If you don't want the action query to execute, you can set the action argument in the event handlers and call AcceptChanges yourselfone of the only times it makes sense to call AcceptChanges.

Stepping Through the UpdateEvents Example

I wrote an example UpdateEvents to illustrate an application that performs many of these functionsyou can find it under this chapter on the DVD. This example fetches rows into a DataTable using the DataAdapter Fill method and binds the rowset to a DataGridView. This control handles several jobs besides displaying the data. It also provides a way to change individual columns, add new rows, and delete specified rows. As you'll see later, when there are errors, each row containing an error is automatically flagged.

Let's step through the code. The first routine I want to show you initializes the DataAdapter and its fetch and action commands. Figure 12.15 illustrates how the SelectCommand is initialized to call a parameterized stored procedure, whose @CustomerNameWanted Parameter value is set from a TextBox control on the form. This DataAdapter initialization is done oncebefore the application shows its first form.

Figure 12.15. Initializing the DataAdapter SelectCommand.

[View full size image]

As shown in Figure 12.16, the UpdateCommand calls a stored procedure that executes the UPDATE statement against the specified row (as indicated by the @CustID Parameter value). Pay close attention to how the Parameter objects are declared. Note that the code binds each parameter to the appropriate DataTable column's Original Version value. This way, we don't have to worry about setting the Parameter value just before the Update method is firedit's set automatically.

Figure 12.16. Setting up the DataAdapter SelectCommand.

[View full size image]

The "CustomerUpdate" stored procedure uses TimeStamp concurrency, so it's important to pass the previously fetched (@OriginialTimeStamp) value to the stored procedure. The stored procedure returns the new TimeStamp in @NewTimeStamp in an OUTPUT parameter. Note that I bind the @NewTimeStamp Parameter back to the TimeStamp source column in the local DataTable, which writes the new value back to the "current" version of the TimeStamp column when the Update method succeeds. Setting the UpdateCommand object's UpdatedRowSource property enables this automatic postback data binding. The InsertCommand and DeleteCommand are set up in a similar fashion later in the same subroutine. When the user clicks the "Fetch" button, the following code is invoked to execute the query, populate the DataTable, and bind it to the DataGridView control, as shown in Figure 12.17.

Figure 12.17. Populating the DataTable with the parameterized Fill method call.

[View full size image]

IMHO

Note that I hide the TimeStamp column to help prevent the DataGridView from choking on the binary column. I think it's really dumb that the DataGridView does not have a default behavior for binary values like a TimeStamp. I also added a DataError event handler to deal with these error events.

When the user makes changes to the DataGridView containing the fetched rowset, the DataTable EditMode is set and these changes are posted to the underlying DataTable in behind-the-scenes columns designed to hold interim values or "versions" of the data column values. ADO.NET supports four versions of the data values for each column. You can examine these versions if (and only if) the DataRow.HasVersion property is True. The available versions include:

  • Current Version: Contains the value returned when the Row(n).Item is referencedin other words, the current value of the specified DataRow column. The Current version is exposed via Row(n.CurrentVersion)but is available only if DataRowVersion.Current is True. Figure 12.18 illustrates fetching the Current version of a DataRow. The Current version becomes available when the BeginEdit method is called.

    Figure 12.18. Exposing the Current (and Proposed) versions kept in the Rows collection.

    [View full size image]

  • Default Version: This is the default version of the DataRow valuedepending on the DataRowState. For all states except Detached, the Default version is Current.

  • Original Version: This version contains the pre-modified column value. Its value is set to the Current value when the DataRow or DataTable AcceptChanges method is called.

  • Proposed Version: When a change is made to a column in Edit mode, the value is exposed in the Proposed version. This version is made available when BeginEdit is executed and deleted when CancelEdit or RejectChanges is executed. When RejectChanges is executed, the Proposed version value is replaced with the Current version value.

These "versions" of the DataRow value can be invaluable when you're executing business rules or other editing criteria checks. For example, your code could check to see if the edited value (the Proposed version) is more than X percent higher or lower than the Current or Original version.

As you can see, the DataRow object's BeginEdit, EndEdit, and CancelEdit methods play an important role in how data (especially bound data) is handled within the DataTable. If your code does not properly end an editing session, the data in your bound controls won't (reliably) end up in the bound DataTable.

Executing the Update Method

Okay, we're ready to fire the Update method and get the process of changing the database based on the current contents of the DataTable. In this example, this routine is fairly simplethat's because I'm not using "batch" mode; each action command is executed individuallyusing one round-trip to the server for each operation. Frankly, this approach is far easier to code and handle errors than the batch mode I'll discuss later in this section.

Figure 12.19 illustrates the routine to fire the Update method against the DataAdapter it starts by flushing the bound controls to the bound DataTable (by calling EndEdit). Note that the default DataAdapter.Update syntax updates all of the DataTables associated with the DataAdapterit's a better idea to specify which DataTable in the DataSet to update, as I've done in this example.

Figure 12.19. The Update button event handler.

[View full size image]

Once the Update event fires, it performs several of the steps I outlined earlier in this chapter (see "Using Events to Execute Your Own Update Logic").

Trapping the RowUpdating Event

As each row is prepared to be sent to the server for updating, the RowUpdating event is fired. (The event handler is shown in Figure 12.20.) It fires once for each row being processedbefore the action command has been sent to the server for execution. The event argument e contains detailed information about what's about to be executed and the DataRow in its current (proposed) state. I've programmed this RowUpdating event handler to tackle two jobs:

  1. Process any row-based business rules. In this case, I check to make sure the Discount column is not set to a value outside the range of 0 to N (where N is set by a constant or fetched from the database during application initialization).

  2. Set action command Parameters in situations where the Parameter values are not supplied automatically by Parameter binding.

Figure 12.20. Initializing the DataAdapter RowUpdating event handler.

[View full size image]

I set up the RowUpdating event to make sure that my client-side business rules can permit the row to be updated. In this case, the client-side (application-based) business rules say that a customer's discount cannot be greater than a specified value (75%). Yes, these business rules should be fetched and updated when the application begins and reset when needed by whatever mechanism makes sense.

The RowUpdating event can also be used to set action Command Parameter values. This is not necessary for the UpdateCommand, as I've already chosen to automatically bind these Parameters to the DataTable using the SourceTable Parameter property. Figure 12.21 illustrates how I set the Parameter values for the Insert and Delete action commands and test the business rule(s). Note that the RowUpdating event handler passes the StatementType, which is used to provide special-case code for each action command.

Figure 12.21. Managing business rules and populating Parameter values prior to action command execution.

[View full size image]

If your code's logic or business rules determine that the row should not be passed to the server as an UPDATE or INSERT, or that the row should not be deleted, you have several options. First, you can set the e.Status argument to UpdateStatus.ErrorsOccurred, UpdateStatus.SkipCurrentRow, or UpdateStatus.SkipAllRows. If e.Status is left alone or set to UpdateStatus.Continue, the action command is passed to the server for execution when the event handler ends. Remember, the RowUpdating event fires once for each row that needs inserting, changing, or deleting based on the RowStatethat is, unless you've enabled "batch" mode, which I'll discuss a little later in this section.

Trapping the RowUpdated Event

Once the action command is executed, the DataAdapter RowUpdated event fires. Again, you can take this opportunity to execute business rules, but it's really too late for thatthe rows might have already been added to the database. However, the data state might have been set by another application, so perhaps you'll find a situation where you should throw an error on data validation failure. This is the place to check for errors. As this event fires after each action command is executed, it's fairly easy to handle the errors. Figure 12.22 illustrates how this event handler can be coded. Again, your logic and action commands determine what constitutes an error. Perhaps, the number of RecordsAffected should be six or some other value (other than one). This approach gives you the flexibility to determine what's correct and what's not.

Figure 12.22. Handling the RowUpdated event.

[View full size image]

Managing Batch Mode Updates

When processing changes to a DataTable, unless you enable "Batch Update" mode, each change to the DataTable is treated like a separate operationwith one round-trip to the server for each change. This can be somewhat expensive when there are lots of changes to be made. That said, I wonder how much overhead is too much. Consider that once you switch to batch mode, your exception handlers must be far more complex, and as a result, your application can be harder to write and maintain. If you're finding that these round-trips are affecting performance, perhaps it's because you're making bulk changeschanges that would better be handled on the server or with the SqlBulkCopy method.

Enabling batch mode is deceptively simpleall you do is change the DataAdapter UpdateBatchSize property to a value greater than 1. The UpdateBatchSize determines how many rows are sent to SQL Server in a single round-trip. If it's set to 0, ADO.NET sets the size to the maximum size supported by SQL Server. In my experience, a batch size of about 50 is optimala larger batch size overloads the caching algorithms, a smaller size defeats the purpose of batching. Of course, your mileage may vary, so experiment with various batch sizes to see which works best for you.

I say "deceptively" simple because once you set batch mode (which is enabled for the first time in ADO.NET 2.0), several criteria change. Let's step through these changes:

  • In batch mode, the UpdatedRowSource property (which determines how [or if] ADO.NET fetches the post-update values from your UpdateCommand and InsertCommand) must be set to None or OutputParameters. This means, in batch mode, ADO.NET can't accept rowsets from SELECT statements executed after the action command.

  • In batch mode, the RowUpdating event fires once for each row being processed, just as before, but the RowUpdated event fires only after all of the rows have been processed. The event handler returns an array of DataRow objects that contain the error status and RowState for each row processedthis is fetched with the e,CopyToRows method, as shown in the example (illustrated in Figure 12.23).

    Figure 12.23. The btnUpdate_Click event handler invokes the Update method.

    [View full size image]

  • In batch mode, you can choose to stop processing the batch on first error or continue despite errors. If you choose to stop on the first error, no changes are posted to the databaseeven when 1 change out of 100 fails. If you choose to stop only after all change operations have been attempted, you can leave your database in an indeterminate statewith some changes made and others not.

  • In batch mode, the entire batch can be rolled back if the severity of the exception thrown by SQL Server in the procedure (or in a trigger) exceeds Severity 11.

Examining the ExecuteUpdateEvents BatchMode Example

I modified the previous example to incorporate batch mode processing, and as I said, things got far more complex. Let's visit some of the parts of the application that were affected. Let's start with the btnUpdate_Click event handler (shown in Figure 12.23) that is fired when the user requests to post changes to the database.

While this routine is similar to the earlier example, note that I set the DataAdapter UpdateBatchSize and the DataAdapter ContinueUpdateOnError properties before executing the Update method. This code is set up to handle exceptions from at least three different sourcesfrom SQL Server (SqlException) and from ADO.NET (DBConcurrencyException and DataException).

Let's continue with the RowUpdated event handler, shown in Figure 12.24.

Figure 12.24. The RowUpdated event handler for the batch mode example.

[View full size image]

Notice that the SqlRowUpdatedEventArgs structure exposes a CopyToRows method that returns an array of DataRow objects. These objects contain the RowState, Table, HasErrors, and RowError properties of the DataRows processed in the last batch. The example binds this array of DataRow objects to a DataGridView to show what it contains. In this case (as shown in Figure 12.25), it does not contain any indication that any row failed to update.

Figure 12.25. The CopyToRows method returns the RowState and other details of the action command operations.

The code in Figure 12.24 also dumps the list of errors for each column and row in this DataRow array populated with CopyToRows. The rest of the RowUpdated event handler is basically the same as the nonbatch mode example shown earlier.


Managing Concurrency by Design or Collision

Managing concurrency is one of the toughest aspects of making changes to any database, whether it's a 3 x 5 card file or a 40TB server farm supporting an ecommerce site selling used fruit. The struggle is deciding what to do when more than one user has update access to the database. Over the years, I keep coming back to a few fundamental solutionsmost of which have to do with designing systems where multiple access is prevented in the first place. Sure, there are systems where this approach is simply not possible, but on closer inspection, the developers (and I) often find reasonable ways that this "prevent collision" strategy can be implemented. If this is possible, it makes your collision handlers far simpler.

One approach that implements this strategy is the "no insert" paradigm. I've also called this the "waitress" or "anvil salesman's" technique over the years. Imagine a restaurant where the waitresses use pre-numbered pads to take orders so no two waitresses in the restaurant have the same range of numbers. This is a "disconnected" database of sorts. It requires no connection to the restaurant system until the order is actually entered in the systemand then only momentarily. Collisions are not possible unless waitress Sally tries to change waiter Sam's chicken-fried steak order to liver and onions. Again, this is a clear violation of the system rules and could be handled with a clear system policyno one is permitted to order liver and onions.

To implement this and similar schemes, set up your system so applications (users) "check out" a set of rows to update and post at a later timeeven days or months in the future. When the new rows are created, the identity values are assigned by the server in bulkestablishing the "ownership" of the rows. Typically, the application calls a stored procedure that creates a block of pre-filled Customers rowsenough to last until the application is connected again. These new Customers rows are fetched into application memorythey already contain unique PKs or identity values and default settings that reflect the users/application's attributes. The connection need not remain open any longer than it takes to fetch the new rows (the new receipt pad). If the application ends and the list of new rows is lost, a fresh list could be refetched from the server or reloaded from a client-side data store (possibly persisted in a SQL Server Everywhere database).

In a similar fashion, your application can call stored procedures to create pre-populated Orders and Items rows ahead of time and fetch these into application memorythese can be based on one of the Customers rows you've already checked out, but that's not really necessary, as the ownership chain starts with the parent (Customer) rows. This means the application could create new Orders and Items rows and set the PK with confidence that the relational hierarchy would be correcteven when persisted to the server database. These could also be persisted locally (a SQL Server Everywhere or SQL Express database might be a good choice for this). Because your application "owns" these uncompleted rowsthe server has them checked out to your applicationyou don't have to worry about collisions. When reconnected, your application need not worry about some other application altering your rowsand you can't change theirs. Sure, your database might have to accommodate some "placeholder" rows, but given the capacities of today's databases (and hard drives), this seems far less of a concern now than when databases needed to fit on a 10MB drive.

Managing an @@Identity Crisis[3]

[3] In April 2003, I wrote an article ("Managing an @@Identity Crisis") for MSDN magazine that discussed how to set up and manage relational database tables where the primary key was implemented through use of an Identity column. This section excerpts and updates that article.

As some of you know, I spend a portion of my day trolling the newsgroups, fishing for interesting topics. Nowadays, I stay away from JET/Access questionsI'm no longer impartial enough to answer these folks without putting a knot in my stomach. I keep thinking, "You wouldn't be asking this question if you weren't using a toy database engine." But that's fodder for another article. Each week, there are several questions that discuss how to handle new Identity values. For example, once you add a new row to a database table that includes an Identity column, how can you tell what value the server assigned to the Identity column?

This section further illuminates the mechanics of managing a system that depends on Identity column update strategies, as illustrated by the examples shown earlier in this chapter. I call this strategy "Post-INSERT fetching," as it executes an additional SELECT query that returns the newly created Identity value after the INSERT is executed. This SELECT can return a rowset or OUTPUT parameters to ADO.NET, which (if cajoled) can be used to post the newly created Identity value back into the client-side row. Both of the examples just shown in this chapter implement this strategy. But there's more to handling Identity values than simply fetching the newly generated valuethat's actually fairly easy (unless you're using the CommandBuilder, as I also discussed earlier in this chapter). What about the "identity" values for child rows that are created on the client? How should you manage these values when you're creating parent-child relationships and need to have valid Identity values to manage inter-table relationships? We'll discuss that, too.

What Are Identity Columns?

Before we wade into the swamp of Identity details, let's clear up a few concepts for those not up-to-speed on the fundamentals. An Identity column is used to provide a unique Integer value that's guaranteed to be unique in the table within the scope of the serverand no further. That is, if you have several servers spread all over the world, there's no guarantee that Identity values generated for the CustomerOrders table in Boston won't collide with the values generated for the same table in another identical database in Cleveland or Calcutta. This means that if you want a unique number that's guaranteed to be unique worldwide, you can't use an Identity without a further qualifier (such as a system ID). In this case, you should consider use of a GUID UNIQUEIDENTIFIER instead. For the context of this section, let's just assume that we're working with a single DBMS server and don't care about replicating with another server's database.

How Does SQL Server Manage Identity Values?

When you execute INSERT against a table row containing an Identity column, the T-SQL does not include a value for the Identity column because the DBMS server automatically adds an increment (usually 1) to the highest Identity value in the table and uses this value for the new row's Identity value. SQL Server also saves the value in a connection-global variable: @@IDENTITY. This means that if an INSERT statement (however it's executed) causes a trigger to fire that also adds a row to a table, the @@IDENTITY value is set to the new value. That's fine (unless your database uses triggers), but what happens when a row is deleted? Is the deleted row's Identity value forever orphaned? Yep, unless you reseed the Identity (DBCC CHECKIDENT), deleted Identity values are lost. Identity values are also orphaned when a transaction is rolled back. This means that when you use Identity columns, you'll need to be prepared for gaps in the series. It also means that eventually, the Integer you're using will overflow, so it's important that you use an Integer datatype large enough for your needsnow and well into the future. The "integer" datatype in SQL Server can identify about 2 billion rows, while a "bigint" can identify 9,223,372,036,854,775,807 rows (that's a lot of rows). However, a "smallint" can identify only about 32,000 rows. I actually had someone complain that he ran out of Identity valuesthey had used a "tinyint", which ran out after 255 rows. Sigh. I'm not going to delve into techniques to recover orphaned Identity valuesit's tough to do, and over the years, I've found it's not worth the trouble. Just make sure to define an integer wide enough to get you through the next century or corporate takeover when they rewrite everything anywayperhaps they'll bring you out of retirement to fix it.

How ADO.NET Handles Identity Values

ADO.NET has its own mechanism to handle client-side Identity values because ADO.NET works with "disconnected" data and does not expect to have live access to the "real" server-side data table. This means that as you add rows to a DataTable object on your client, the Identity value generated locally by ADO.NET won't have any bearing on the Identity values of existing rows in the database or on the rows in the local DataTable. Huh? How can that work? Well, ADO.NET does not make any effort to test the Identity values it generates against existing Identity values in the client-side, disconnected DataTable. It sets the new Identity values based on the AutoIncrementSeed and AutoIncrementStep. This means that if your existing DataTable has Identity values ranging from 1 to 10 and you set the AutoIncrementSeed value to 10, ADO.NET does what it's toldit starts at 10 and you'll likely end up with two rows with the same Identity value (not good). These "autoincrement" properties can be set before you add any rows to the DataTableafterward, it does not seem to matter. The demonstration application included with this section illustrates this behavior.

"Tricking" ADO.NET

When setting a client-side Identity value, the trick is to set a value that the server-side database table is not using or not likely to be used by rows being added by other clients. That way, any new rows won't collide with existing rows in the current DataTable. Since server-side Identity values are invariably positive integers, you have only one other set of numbers to use on the clientnegative numbers. ADO.NET is ready to handle this contingencyset both the AutoIncrementSeed and the AutoIncrementStep to 1. This way, each new row is created with an Identity unique to the client. This also means that a row in a parent-child relationship can easily identify its relations before the server assigns their "real" Identity value when you execute the INSERT query. These negative numbers are used to interrelate parents with their children until ADO.NET inserts the new rows into the server-side database with the Update method. Remember, the INSERT statements you or ADO.NET generates (via the CommandBuilder) don't include the client-side Identity valuesthey're needed only to identify client-side rows and relationships.

Updating Parent-Child Relationships

Once you create a new parent row (for example, a row in the Customers table) and let ADO.NET generate its new client-side Identity value, you can create as many child rows (for example, Orders) as necessary and safely use the parent's ADO.NET-generated Identity value as a foreign key (so the child row is related back to the correct parent). Yes, ADO.NET knows how to handle these relationships correctly when it comes time to post these new rows to the server. When you execute the Update method, ADO.NET executes the INSERT for the parent row first and then all associated child rows. If you set up your InsertCommand correctly, the server-generated Identity value is propagated to the child foreign key valueI showed you how to do this in the previous section.

This sample application did not avail itself of any of the client-side hierarchy-management features (if you can call them that) in ADO.NETthere was no need.

Figure 12.26 illustrates the routine that walks through the hierarchy used in this application. In this case, it's the typical Customer, Orders, Items hierarchy. The Customer table is a parent to the Orders table, which is a parent to the Items table. Because of this three-tiered approach, we must perform the updates in a specific sequence to maintain referential integrity. In other words, this sequence means that no child is created without an immediate parent, and no parent is deleted if it has an immediate child. Since there are three levels of hierarchy here, the Orders table is both a parent (to the Items table) and a child (of the Customers table).

Figure 12.26. Posting changes to a hierarchy of DataTable objects.

[View full size image]

To implement this hierarchical update:

  • The first time Update is invoked, any new rows are added and any changes are made to the Customers tablerows marked for deletion are left in the DataTable.

  • The second call to Update repeats the add/change process for the first-level child table (Orders).

  • The third call to Update posts all adds, changes, and deletions to the third-level child table (Items).

  • The fourth call to Update deletes any second-tier parent rows marked for deletion (Orders).

  • The fifth and final call to Update deletes any first-tier parent rows marked for deletion (Customers).

This way, children are added after their parents but are deleted only after the parents are deleted. The last updates delete any parent rows marked for removal, so the children are deleted first (before their parents).

Nope, you don't see a call to the AcceptChanges method in this codeit's not necessary, as it's called automatically by the Update method after it posts its changes to the database.

Retrieving New Identity Values

The real problems come when you want to find out what Identity values were generated by SQL Server. No, ADO.NET does nothing on its own to help. Visual Studio's DataAdapter Configuration Wizard (DACW) can help generate additional SQL for your InsertCommand to retrieve the new row's Identity, but the CommandBuilder is clueless in this regard, so it's no help at all. In the example I wrote for this section (see "Managing Identity Hierarchies" on the DVD), I used the DACW three times to build SqlDataAdapter objects for each of the three database tablesCustomers, Orders, and Items. I then created DataSet objects for each of these using the Generate DataSet "button" in the DataAdapter Property dialog. This can be referenced by right-clicking on the tooltray icon created for each SqlDataAdapter and choosing "Properties"as shown in Figure 12.27.

Figure 12.27. Creating a new DataSet from the DataAdapter Properties dialog.

These DataAdapters were created to invoke stored procedures to fetch and perform the action commands. The InsertCommand stored procedure (as shown in Figure 12.28) includes a SELECT to fetch the newly created Identity value. ADO.NET fetches this automatically, as discussed earlier in this chapter. In this case, the value is returned as an OUTPUT parameter, but ADO.NET also recognizes a rowset containing the new Identity value.

Figure 12.28. Fetching the newly created Identity value.

[View full size image]

Problems with @@ IDENTITY

The example I wrote for this section did not depend on any code generated by the CommandBuilder or the DACW. I was worried that the problems mentioned in 2003 had not been addressed. To circumvent the issue, I simply pointed the DACW to my own stored procedures that fetched the rows and performed the action commands. However, when I first wrote about this issue, the DACW-generated code included a reference to @@IDENTITY, the global variable I mentioned earlier that contains the last Identity value generated on the connection. In the current version of Visual Studio, this problem has been addressed (I guess someone read my article)the generated code uses the new (implemented in SQL Server 2000) SCOPE_IDENTITY() function that returns the Identity value created in the programmatic scope of the procedure that inserted the new row.

If you find code that still uses @@IDENTITY, do yourself a favor and replace it with SCOPE_IDENTITY(). That's because use of @@IDENTITY assumes that your database does not expect any triggers to fire when the INSERT is executed. If a trigger fires after you insert a new row, and if that trigger adds another row to a table (any table that has an Identity column), the @@IDENTITY global variable is set to that new Identity valuenot the one your INSERT generated. This makes the V1.1 DACWgenerated code work for simple situations, but not when your database gets more sophisticated. Hopefully, that won't happen until after you retire.


Summary

Yes, there are a few pieces left out of this chapter that I included in my earlier book ADO and ADO.NET Examples and Best Practices for Visual Basic Programmers, but I've given you all of the essential ingredients to help you set up and manage changes for your database. I didn't want to repeat that information here. I have discussed how to use the TableAdapter (and DataAdapter) wizards to create updateable DataTable and DataSet objects, as well as how to customize these to meet the needs of your application architecture. In all of my examples, I tried to avoid use of the CommandBuilder (as most other books take that route) and emphasize designs that are more like the ones your DBA will insist uponthose that leverage stored procedures and scalable architectures. All of the stored procedures used here and elsewhere in the book are in the Biblio or one of the other half-dozen or so sample databases that are included on the book's DVD. I encourage you to explore these procedures. You'll also discover that the Biblio database (and especially the Customers table) has a trigger that demonstrates how to implement a simple business rule in T-SQL server-side code.

I expect to revisit this subject in one or more EBooks as well.


Chapter 13. Managing SQL Server CLR Executables

    Introduction

    Evolving CLR Executables Technology

    Where Do CLR Executables Make Sense?

    Designing a CLR Executable

    Building Your First CLR Executable

    Building and Deploying a CLR Executable

    Setting Up a Test Environment

    Working with ADO.NET from Within CLR Executables

    Coding CLR Stored Procedures

    Implementing Basic User-Defined Type (UDT) CLR Executables

    Implementing Advanced CLR User Defined Types

    Accessing CLR UDTs Elsewhere

    Implementing CLR Aggregate Executables

    Implementing CLR Triggers

    Advanced Debugging

    UDT Safety and IP Security

    Summary


Introduction

SQL Server CLR executables are probably the most widely anticipated and possibly the most widely misunderstood of the new Visual Studio and SQL Server 2005 features. Over the last few years (long before the final release of SQL Server 2005), every developer on this side of the galaxy was exposed to magazine articles, technical conference sessions, and street performers that detailed how the .NET Framework's Common Language Runtime (CLR) was to be incorporated into SQL Server. As the sessions extolled, for the first time, stored procedures, triggers, functions, user-defined datatypes, and aggregates could be written in a non-T-SQL language. Based solely on the enthusiasm of the presenters, some developers left the sessions early so that they could get back home to rip out all of their existing stored procedures and replace them with C# code. We all have been cleaning up the mess ever since.

On a more conservative note, I wrote my share of this content in an attempt to return sanity to these presentations and portray the new CLR coding feature in a more realistic, down-to-earth light. It seems that others (and too many from Microsoft) could not dampen their unbridled enthusiasm for stored procedures written in C# or Visual Basic .NET. Those leaving these Microsoft sessions were still convinced this technology could solve all of the world's problems, including global warming and the war in the Mideastbased on the current state of the planet, I don't think it's working.

As I'm about to demonstrate, CLR executables can't lower ocean levels and they aren't a good substitute for your run-of-the mill stored procedures and T-SQL functions. This chapter tells you why. Don't get me wrongthere are excellent reasons to leverage the power and performance of the .NET Framework and CLR executables, but you'll find that not all situations call for them. In this chapter, I'll discuss the hows, whys, and whens of this new SQL Server CLR executable technology. For those cases where CLR executables make sense, I'll show you how to design, code, debug, and deploy a number of executables. All along the way, I'll compare performance among various approaches and show you situations where a CLR stored procedure or function can make a positive difference and those where it just gets in the way. We'll see how DBAs can control this new feature (to an extent) and how developers must gain the trust and rights from DBAs to get access to the technology.

Can Visual Basic .NET Developers Play?

A senior data-access manager at Microsoft once suggested that CLR executables were not really for "Visual Basic" programmershe implied this new technology was too complex for them to grasp. I (and others) took umbrage at this remark and nearly rushed the stage with drawn weapons. We made it abundantly clear to this "expert" that many Visual Basic developers were fully capable of dealing with the most complex issues Microsoft could throw at them. That said, I also agreed that some developers (regardless of their language of choice) were not ready to dive head-long into the complexities of SQL Server CLR executables. I understand that many application developers are not SQL Serveraware or competent DBAs. Yes, I expect that the discussions in this chapter are possibly some of the most complex in the entire book. I suggest you read it anywayregardless of your skill levelespecially if you plan to use SQL Server CLR executables anytime in the future. In addition, after having read this chapter, I expect that virtually any developer will have enough information to really screw up their production SQL Server.

What Role Can Visual Studio Play?

For reasons I fully understand (but do not excuse), many magazine articles and MSDN help topics on this new technology act as if Visual Studio 2005 does not exist. They walk through the fairly laborious process of code, testing and deploying the CLR executables via command-line and SQL Enterprise Studio queries. They must think it's more macho to not use the tools Microsoft has made available. I suspect that those writing these articles work strictly with SQL Server and don't lower themselves to work with Visual Studio. Ironically, it turns out that Visual Studio 2005 has eliminated much of the grief we all face when working with CLR executables. It actually makes development easiereven dangerously easy. I'll be sure to show you the easy and safe ways.

Just one more point on Visual Studio 2005. Since I wrote this chapter, I've spent countless hours working with the design, coding, deployment, and debugging steps needed to create over a dozen CLR executables. Yes, Visual Studio 2005 works better than ever before. Given that I've worked with the earliest (pre-public) versions of Yukon and Whidbey, I've also spent countless hours trying to help Microsoft make these products work well together. The end-result is pretty goodbut just that. I endured any number of incidents where Visual Studio 2005 locked up, got stuck in the mud, or rolled over in the ditch smoking with its wheels in the air, tossing hours of coding into the ditch as it crashed. No, Visual Studio 2005 is far from perfect, but even with these problems, it was really pretty easy to create most CLR executables. This does not include User-Defined types (UDTs), as Visual Studio does not know how to let one debug a CLR UDT without reinstalling it firstwhich is not an option due to dependency issues. Yes, Microsoft is still working on these issues. Let's hope they figure them out soonhopefully without reinventing the wheel (again)s.

IMHO

Making it easy for developers of all skill levels to screw up their production servers with ill-conceived CLR executables is not such a good idea.


Evolving CLR Executables Technology

In earliers versions of SQL Server, developers, DBAs, and code-generator tools coded stored procedures, triggers, functions, and other objects in T-SQLand just T-SQL. However, if your skills are sharp enough, you can develop and call an Extended Stored Procedure (ESP) written in C or C++. Not many developers choose to take the ESP route, as it requires a thorough understanding of Open Data Services (ODS), COM, and other complex (and potentially destructive) technology. Not only that, but developers need to get their DBA to sign off on their use, and DBAs are naturally hesitant to do so. That's because if the ESP code screws up, it can (easily) bring down the server or worse[1], and you'll need someone to take the blame.

[1] For an excellent article on Extended Stored Procedures, check out this article by Kevin Cox, published by 29th Street Press (1997): www.windowsitlibrary.com/Content/77/20/1.html#1

ESPs did not gain much traction back in the late 1990s, when they were first discussedthey were seen as too hard to code and maintain and far too dangerous. Understand that an ESP could kill SQL Server in a heartbeat. It's kinda like juggling with chainsawscool to watch until something goes wrong. As a result, ESPs were used very selectively to address some of the most CPU-intensive tasksthose well beyond the reach of T-SQL. For example, ESPs were implemented to perform complex geometry, trigonometry, or statistical calculations; expose low-level Windows API functionality; perform other complex CPU-intensive operations; or just provide functionality that T-SQL could not perform without assistance or a lot of code. Over the years, the SQL Server team exposed a number of system ESPs for use by the SQL Server tools and special DBA functions.

One problem with using ESPs is that SQL Server must (by definition) fully trust the codethere is no "partial-trust" or "sorta-trust" option. "Trust" functionality is first enabled with SQL Server 2005. Because you use unmanaged code to create ESPs and they run in the same address space as SQL Server, a problem in an extended stored procedure could damage SQL Serverin fact, it could blue-screen the system. Not good. Now who was it that said we should use ESPs?

Are CLR executables any safer than the ESP procedures they replace? No, not necessarily. If the CLR executable has to be marked as "Unsafe", it can do just as much damage as an ESP executable.

Tip

Microsoft wrote significant parts of SQL Server 2005 using CLR languages such as managed C++, C#, and Visual Basic .NET.

Microsoft's ESP Replacement

In SQL sServer 2005, T-SQL is still the language of choice for server-side executables, but you can also implement many of these executable objects in C# and Visual Basic .NET, and whatever .NET Framework CLR languages Microsoft chooses to add. In a nutshell, SQL Server 2005 exposes .NET Framework functionality by incorporating the ability to execute CLR code from within the SQL Server address space. This gives developers the option to:

  • Write SQL Server executable code such as stored procedures, functions, and triggers in CLR-based C# or Visual Basic .NET, and invoke those stored procedures just as T-SQL stored procedures are executed

  • Write CLR-based user-defined data types (UDTs) and aggregates, and use them and their associated code to define your tables and business rules

  • Execute CLR code in three defined levels of trust"SAFE" (completely untrusted), "EXTERNAL_ACCESS" (partially trusted), and "UNSAFE" (completely trusted)[2]

    [2] These characterizations of the trust level might seem backward, but they aren't. See the further discussion later in this chapter.

  • Write replacements for ESPs that perform the same operations, but (potentially) more safely and with direct access to the latest SQL Server datatypes

The real question is: Should you use CLR executables? If you remember Jurassic Park (1993) and Jeff Goldbloom's character's speech to Mr. Hammond:

Yeah, but your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should.[3]

[3] Quoted from the movie Jurassic Park. © 1993, Universal Studios.

No, I don't think that CLR executables are reincarnations of carnivorous velociraptors or lumbering brontosauruses in disguise, but this technology is a radical departure from well-known and well-understood T-SQL and should be treated (like any new technology that I'm just learning) with a good measure of respect. I also hope you have doubly redundant high-tension fences placed strategically around your serverjust in case.


Where Do CLR Executables Make Sense?

Consider that the code SQL Server executes for each connection is only one of many jobs/tasks/threads the server is expected to execute. SQL Server is and always has been a "database" server designed to locate and store data managed by remote clientseven if they are on the same system. Generally, the more code SQL Server has to execute, the fewer operations of all kinds it can execute in a given length of time. SQL Server has also been a host for its own brand of executablesfrom the earliest versions, SQL Server has supported T-SQL stored procedures.

Sure, there are some tasks that make sense to execute on the serverespecially when the operations require direct (and proximitous) access to the data. For example, if you execute a query that returns all airports close to a specified hotel based on GPS coordinates, you'll probably end up performing quite a few geometric equations that are a bit difficult for T-SQL code. You might try to move all of the pertinent data to the client and perform the calculations in an applicationand some applications take this approach. However, the expense of moving too many rows over the wire to a distant client can be prohibitively expensive. My tests show that GPS distance calculations are an ideal candidate for a non-T-SQL code functionyes, a CLR function. However, when working with sets of data (such as those returned by a SELECT statement), calling out to a CLR function from within SQL Server can actually hurt performance. T-SQL and SQL Server work closely together to leverage SQL Server's extensive cache systems to make sure relevant and frequently accessed data is left within easy (and proximitous) reach in memory.

Other factors play a role as well:

  • Does the CLR code leverage the ability to perform recursive or complex operations that cannot be done to sufficient accuracy (or at all) in T-SQL?

  • Is the code being executed already compiled, or must it be interpreted or compiled before first use? This applies to both the T-SQL and CLR routines. Is this a one-time performance expense or a price that must be paid on each execution?

  • Do features built into the CLR language or data access paradigm address problems that can't be easily or directly solved in T-SQL?

  • Are you using a CLR routine because you're not that familiar with T-SQL, or vice versa?

  • Does it take more time to call the CLR routine than it would to simply execute a potentially slower T-SQL routine? In other words, is SQL Server spending more time transitioning to and from the CLR function than actually executing the code?

  • Can the CLR code be written and implemented without having to resort to "fully trusted" security mode where the CLR object can do anything? Can the CLR code function in an "untrusted" or "partially trusted" environment?

  • Does your development team know how to write, test, debug, and ultimately support CLR code well enough to implement the procedures and other objects needed? Are you prepared to support this code indefinitely, or must you bring in consultants to repair or tune it?

  • Have your DBA and management team bought in on use of CLR and the potential security issues it raises? Are they prepared to support the use of CLR code indefinitely? Note that it's tough to go back once CLR executables are enabledit's an "all or nothing" switch on the server instance. It applies to all databases on the server instance.

  • When implementing CLR User-Defined types (UDTs), are you aware of the inability of the tools to automatically change, alter, or drop types and preserve existing data? Do you and the DBA understand the maintenance impact of CLR UDTs?

What Kind of Executables Can or Should Be Built?

To get an idea of where CLR code is possible, let's briefly outline the basic types and provide some insight into how they work, how they're different, and how they can be expected to behave on the server.

CLR Stored Procedures

A CLR stored procedure can accept (virtually) any number of input parameters and return (virtually) any number of OUTPUT parameters, an integer (SqlInt32) RETURN value, and zero or (virtually) any number of resultsets (including 0 to N rowsets). In theory, CLR stored procedures can be used anywhere a T-SQL stored procedure can be used, including "normal" database or master-based stored procedures, DML triggers, or DDL triggers. As with T-SQL triggers, the special virtual INSERTED and DELETED tables are exposed.

CLR Functions

Like a stored procedure, a CLR function can accept any number of input parameters, but it cannot return OUTPUT parameters or resultsetsonly a RETURN value variable that can be (almost) any type, including table-type variables or XML. In theory, CLR functions can be used anywhere a T-SQL function can be used, so they can be called from stored procedures, used in T-SQL expressions, or called from or other functions.

CLR User-Defined types

A CLR User-Defined type (UDT) is a way to build an object that's stored in place of a SQL Server system type in a SQL Server column. Unlike a T-SQL UDT that's simply an alias to another base type, a CLR UDT is a composed of many individual data elements. For example, you could create a "Currency" type that stored not only the monetary value, but also the unit (U.S. dollars, British pounds, Japanese yen, euro dollar), the monetary symbol ($, £, ¥, €), the exchange rate, the base currency and any other pertinent information. Consider that as the complexity of the UDT increases, its usefulness and performance decrease exponentially.

IMHO

Consider that as the complexity of the UDT increases, its usefulness and performance decrease exponentially.

Keep in mind that a column definition configured with a CLR UDT can't be easily removed or changed, as with system data typed columns. All references and dependencies to the UDT must be purged from all tables and procedures that reference it before it can be altered.

CLR Aggregates

A CLR aggregate is designed to perform the same type of operations that T-SQL aggregates perform. Examples of T-SQL aggregates are SUM, MIN, MAX, AVG, COUNT, and so forth. These functions accept a set of values and return a single value (ignoring NULL values). CLR aggregates make the most sense when you have to build a custom aggregating operation that's not supported by T-SQL, such as mean, frequency, or other statistical aggregations. One interesting use of a CLR aggregate would be to implement the SUM aggregate for the aforementioned Currency UDT.

CLR Triggers

A CLR trigger is designed to perform the same type of operations that a T-SQL trigger performs. Triggers are your last-ditch firewall of protection on your data tables. They can also manage any number of business functions or implement sophisticated business rules. SQL Server 2005 adds DML triggers to the mix, which means you can create triggers that monitor changes in the tables, users, or other objects (even other triggers). A trigger generally implements a business rule that tests the new (INSERTED) or updated rows about to be committed to a specified table. If the rule finds there's something wrong, it can choose to roll back the transaction. A CLR trigger is no different, except that it can execute far more sophisticated tests, as it's not hamstrung by the limitations of the T-SQL language. Later in this chapter, I illustrate a CLR trigger that executes one of the Framework's financial functions, just to illustrate this point.

Previewing the Examples

Once you've read this chapter, you'll get a pretty good idea of the CLR executable solutions that make sensethose that outperform the T-SQL they're replacing. The examples implemented in this chapter include:

  • A set of CLR functions to convert temperature values (actually, an example of what not to do)

  • A CLR function to return a Table-type variable generated from a delimited string

  • A CLR function to count the number of times a string appears in another string

  • A CLR function to compute the distance between two points of latitude and longitude

  • A CLR UDT to manage international currency data

  • A CLR stored procedure to count specific words found in a TEXT (varchar(max)) column

  • A CLR aggregate to sum values in the international currency UDT columns

  • A CLR trigger that accepts or rejects additions to a table based on the results of a financial analysis of new or changed row data

Along the way, I've done performance tests on these carefully chosen solutions to see which make sense. All but the first (temperature conversion) performed as well as their T-SQL counterparts (when the problem could be solved with T-SQL). I also came to some other conclusions:

  • When T-SQL has to perform string functions, CLR can usually outperform T-SQL. This includes searching, concatenation, and other bulk text operations. This is evidenced in several examples where I manipulated strings.

  • When T-SQL has to perform trigonometry, financial, or higher math functions, CLR does better or at least as well. The calculations I performed were complex for a sixth-grade math student (at least, in the U.S.) but not that hard for T-SQL. I also suspect that there are many situations where far more complex math is required, as when doing financial or statistical calculations or aggregations. In these cases, CLR code would certainly perform better, as T-SQL has no support for these operations.

  • In situations where the client must draw data from the server, munge, and return it, a server-side CLR executable can easily outperform the client executable. All too often, we've all seen developers pull heavy rowsets from the server, update a row based on a complex algorithm, and return the result to the server via an UPDATE. Part of the problem here is that many developers are not familiar with T-SQL and don't appreciate that the task can be more easily performed on the server. By implementing a CLR executable in a familiar language, these developers can solve a problem they could not even approach with T-SQLnot because of T-SQL limitations, but because of their lack of T-SQL skill.

  • Not all CLR and Framework functionality is available on all types of CLR executables. For example, an aggregate cannot use the local "context" connection to capture data. As we progress through the types of executables, we'll witness many other limitations.

  • When invoking functionality available only via .NET Framework classes, you don't have a T-SQL optionyou must forgo the functionality or code it in a CLR executable. For example, if you want to encrypt or compress data, use the financial functions, or access system internals, the Framework has built-in functionality that supports these operations that can't be accessed directly from T-SQL (at least, not without an ESP).

This should give you a rough idea of what works and why. After you've worked through these examples, you'll find how well (or if) CLR executables can solve your problems.

What Makes CLR Executables Fast and Safe?

If you've chosen the right job for a SQL Server executable, it can outperform a T-SQL executable or perform a task that T-SQL simply can't handle. SQL Server 2005 plays an important role in the performance of a CLR executable. One factor that helps performance is that the CLR executable is managed code that runs within the database service process. This is possible because the .NET Framework (thus, the CLR) is hosted within SQL Server's process space. This makes transitioning between SQL Server (T-SQL) and the CLR executable more efficient and permits SQL Server to manage cross-operation conflicts and integrate the memory demands of the CLR executable with the demands of other operations it's managing. To make this work, a CLR executable requests memory from SQL Servernot Windows. As with other SQL Server internals, like the query processor, SQL Server manages memory requests from a common cached pool of memory that it orchestrates based on frequency of use and other logic-related factors. Windows would have no idea how important a particular block of memory is in regard to your query or CLR executable. Because garbage collection is handled by SQL Server, this CPU-intensive operation can be coordinated with other SQL Server operations to optimize performance and idle time. As I'll illustrate later in this chapter, a CLR executable in need of data rows can access a local "context" connection that completely bypasses the network protocol and TDS translation layers. All of these features taken together also prevent a CLR executable from running away with the CPU or compromising the stability of SQL Server.[4]

[4] For more details, see Kimberly Tripp's article "The Database Administrator's Guide to the SQL Server Database Engine .NET Common Language Runtime Environment" (www.only4gurus.net/miscellaneous/dba_guide.pdf).

Tip

Remember that CLR executable support is "off" by default in all versions of SQL Server. It must be enabled before it can be used.


Designing a CLR Executable

Before you rush off to create your first CLR executable, it makes sense to pause and think a moment about how they work and what's involved behind the scenes to support their execution in SQL Server. One way to think about CLR executables is to handle them as you would handle remote COM (DCOM) components. Anyone who has worked with a remote component learns that it's expensive to make the trip to the component executable code, so it makes sense to do as much work as possible in the remote component once you arrive. That is, while remote components can expose public properties, it doesn't usually make sense to implement them because it costs too much to call the remote component just to set one static variable. This also assumes the component is designed to maintain state between property setting calls. This is also problematic for any system that hosts these objects, as they have to manage multiple instances of the objecteach with its own state.

On the other hand, what developers usually do when working with remote objects is execute methods on the remote objectspassing in "property" settings as method arguments. This reduces the number of round-trips, simplifies the design, and permits the invoked code (and the host) to create and managed stateless objects. This yields far more scalability. When you start calling CLR executables, keep this concept in mind. Design your components to support rich methods and few to no "initializers" or "destructors."

When the CLR executable is first executed, SQL Server must load a set of DLLs into memory so your first encounter with a CLR executable is expensive. Once in place, these DLLs remain in memory (competing for space) until RAM is needed by other competing processes. The first time your executable is invoked, the just in time (JIT) CLR compiler creates a processor-specific block of code to execute itagain impacting first-run performance. Again, this machine executable code is left in memory until all instances are done and the space is needed for other operations.


Building Your First CLR Executable

Let's get right down to it. Assuming you've decided that you have a routine that you think can be implemented more efficiently by calling a CLR executable, you're ready to step through the process of implementing it using Visual Studio. This example walk-through builds a pair of CLR functions: one to convert Fahrenheit temperature values to Celsius and another to convert back to Celsius from Fahrenheit. Sure, these routines can be built with T-SQL. The purpose of this exercise is to show you how much performance can be gained (or lost) by using a CLR function.

IMHO

Overheard in the hallway: "I'm not a DBA, but I play one at work...."

1.
Before you get started wizarding and coding, you'll need to ask permission from your DBA to enable CLR executables on the target SQL Server. If you're acting as the DBA, you need to ask yourself the important security questions all DBAs ask. I listed many of these earlier in this chapter. If you get permission, start the SQL Server Surface Area Configuration utility that ships with SQL Server 2005. You should see an initial dialog that looks like Figure 13.1.

Figure 13.1. Choose the target server with the SQL Server Surface Area Configuration tool.

[View full size image]

The process of enabling CLR executables can also be done via SQL commands, as shown in Figure 13.2. Regardless of the technique you use, this setting applies to the server instancenot just to a specific database. Once it's enabled and your developers start implementing CLR executable functionality, you'll find it's tough to go back and unwind all of the functionality these executables expose.

Figure 13.2. Using a SQL Server Management Studio batch to enable CLR executables.

[View full size image]

2.
Next, click on "Surface Area Configuration for Features". This populates a tree view for each SQL Server instance running on the target machine, as shown in Figure 13.3. Select "CLR Integration" and make sure the check box is selected. Click OK to continueit's off by default. If you're working with a team, it's important to leave this feature on once enabled, to permit other developers' CLR executables to continue working.

Figure 13.3. Enabling CLR integration on the target SQL Server instance.

[View full size image]

3.
Next, fire up Visual Studio, start a new project, and choose your favorite language. Drill down into the "Database" project types. Nope, while one might think so, this template does not come from the list of "Business Intelligence" types. You should see one (and only one) project type here: "SQL Server Project", as shown in Figure 13.4.

Figure 13.4. Creating a new CLR executable project and solution.

[View full size image]

Note that I set the project name to have a "prj" prefix and the solution name to have an "sln" prefix. This naming convention helps when it comes time to see how these are deployed to SQL Server. Click OK to continue.

4.
Next, Visual Studio exposes a dialog to permit you to choose a "Database Reference" from one of the registered Data Connections (as managed by the Server Explorer). Basically, all you're doing here is setting a ConnectionString to address the SQL Server instance and database that's intended to host the CLR executable. If the desired Data Connection is not in the list (as shown in Figure 13.5), you can click on "Add New Reference..." to create another Server Explorer-managed Data Connection.

Figure 13.5. Choose one of the exposed Data Connections.

It's this Database Reference (Data Connection) that's accessed when it comes time to deploy the stored procedure and to address the host (a specific SQL Server instance database) when debugging. It can be changed later (as when switching from testing to production) by changing the Connection String setting in the project properties (Database tab). I discuss this project property and others a bit later.

5.
If you haven't done so already, Visual Studio warns you that the specified connection must support SQL/CLR debugging if you want to use this feature. Yes, this is different than supporting the ability to host a CLR executable, as I discussed earlier. It's a good idea to click "Yes" on this dialog if you intend to debug your code.

While you're debugging your SQL Server CLR executable, all (not just your) CLR executables (managed threads) are stopped. This means if you're testing on a production server, the DBA might come and stand in your office eyeing your network cable (or your neck) with an axe.

6.
To add a CLR executable class template to your project (regardless of the type of executable you want to build), right-click the project in the Project Explorer and choose "Add", which exposes another dialog (shown in Figure 13.6). Pick any one of the CLR executable templatesit makes no difference at this pointyou're just going to be taken to another selection dialog where you can make your final choice.

Figure 13.6. Choosing a CLR executable template.

7.
Since I'm building a CLR Function to convert Fahrenheit to Celsius (and back), I choose "User-Defined Function...". This opens the "Add New Item" dialog, where you're prompted to choose one of the Visual Studio installed templates (including templates of your own) and name the new Function. Note that I named the Function generically (as shown in Figure 13.7). That is, this clsTempConversions.vb class is designed to contain several "conversion" functions whose names are determined by code within the class. Let's use a unique prefix here as well, so I can (again) track how this code is deployed to SQL Server.

Figure 13.7. Choose the appropriate class file Template and name.

[View full size image]

8.
Each of the templates creates a unique block of code to initialize your CLR executable classspecially configured to correctly interface to SQL Server. Let's take a look at the generated (and unchanged) function template (as shown in Figure 13.8), as it's the easiest to understand.

Figure 13.8. The unmolested CLR Function template.

Note that the default CLR function template (you'll certainly want to create your own) expects no input parameters and returns a SqlString. Note the attribute <Microsoft.SqlServer.Server.SqlFunction>, which tells the compiler that you're building a CLR function. To save time, I'm going to go ahead and modify the generated code to implement the two functions I need for my temperature conversion class.

IMHO

Save often and exit Visual Studio occasionally to preserve your work.

9.
When you use the "function" template, the Public Class created is (always) given the name "UserDefinedFunctions". This class name is not exposed when invoking or addressing the functions in T-SQL, but it is referenced behind the scenes by Visual Studio and the other tools that deploy your application to the target server. No, you don't need to change it unless you plan to have more than one class that defines one or more User-Defined (CLR) Functions.

However, you do need to set the CLR function name and define the function's input parameters. The code shown in Figure 13.9 first defines the function "signature" and is referenced by the applications that call it. Once a function is deployed to the database and developers begin to use it, you won't be able to change this signature without causing more folks to come to your office carrying edged weapons. This is no different than any T-SQL function, stored procedure, or class definition, for that matter.

Figure 13.9. This code defines two CLR functions to convert temperatures.

[View full size image]

Note that both of the publicly exposed functions (fnFahrenheitToCelsius and fnCelsiusToFahrenheit) are also prefixed with an XML attribute[5] that tells the compiler and deployment routines which functions in this class are to be made visible to SQL Server. Sure, the public class UserDefinedFunctions can have any number of other functions or subroutines, but only those properly designated will be exposed to SQL Server. As I step through the other types of CLR executables, you'll see how this attribute is used to define how the executable is accessed, serialized (if necessary), and interfaced.

[5] These compiler-directive "attributes" are called "decorations" in some references.

As with other classes I've written, this class includes a few constants that are visible throughout the class scope. When working with CLR functions, you won't be able to define class-global uninstantiated variables. This means you'll be able to declare Const (in Visual Basic .NET) variables but not other "New" instance variables that require code to run outside the context of the called CLR functions. That's because there's no "New" method for the functions that are called when they're first invoked. Sure, you can declare global variables; you'll just have to initialize them using code within one of the functions. One approach might be to have a "setup" or initialization function that must be called before the others, but that assumes that SQL Server maintains the state of these variables between invocations. It doesn't. What you can do, however, is have the function itself create an instance of a class and invoke it. You still won't be able to retain state between calls.


Building and Deploying a CLR Executable

Once your code is ready to test, you need to double-check the spelling of the function name to make sure it matches the code specification. Since the Visual Studio compilers (perhaps foolishly) assume you know what you're doing, it won't help you spell the entry point of the class you're creating. To build your application, choose "Build" from the project right-click menu or "Build Solution" from the Build menu. This instructs Visual Studio to compile the runtime DLL for the executable. If this succeeds, you're almost ready to deploy.

Preparing to Deploy a CLR Executable Function

The deployment phase installs the CLR executables into the targeted SQL Server database you specified earlier. To view or re-address this database, right-click the Project and choose "Properties". This opens the familiar tabbed dialog (as shown in Figure 13.10) that is the GUI interface for the assemblyinfo(.vb/.cs) configuration file. Click the "Database" tab to view the current Connection String. You can change this manually or click on Browse to re-open the Data Connection chooser dialog you saw earlier in Figure 13.5.

Figure 13.10. Addressing the target database.

[View full size image]

Tip

Sure, it would be useful to visit the SQL Server database to see if the new CLR executable might collide with an existing executable.

Exploring Executables with the Server Explorer

The Server Explorer can be used to view and manage CLR executables, as shown in Figure 13.11. Note that the Server Explorer lists the deployed (and visible) executables along with the Assemblies used to create these executables. Yes, this means the source code used to build your function, stored procedure, aggregate, or User-Defined type is stored in the targeted database.

Figure 13.11. Using the Server Explorer to view (and manage) SQL Server CLR Executables.

On closer inspection, you can see that the Server Explorer categorizes the executables by intermixing T-SQL functions, stored procedures, and types with CLR executables. This can be a bit confusing, especially since all of the function types are rolled together. Note that while the function parameters are shown, you need to drill into the property page to view the data type.

Tip

Missing from this tree view is any reference to ordinary SQL Server User-Defined types.

Exploring Objects with SQL Server Management Studio

In contrast, the SQL Server Management Studio Object Explorer breaks out functions into categories based on the type of data returned, as shown in Figure 13.12. In this case, functions are broken down by "Table-valued" functions (which return a Table-type variable); "Scalar-valued" functions, which return a single value; "Aggregate" functions, which return an aggregated value; and "System" functions, which are implemented by SQL Server. This approach makes SQL Server Management Studio a more appealing choice for developers who need to manage large numbers of executables.

Figure 13.12. Using the SQL Server Management Studio Object Explorer to view deployed executables.

Note that the SQL Server Management Studio Object Explorer also exposes the executable parameters along with the parameter datatype and default. This explorer also lists the CLR "User-Defined Types" (not to be confused with ordinary SQL Server "User-Defined Data Types").

Managing Executable Dependencies

Remember, I'm exploring existing CLR executables to see if my new CLR function will collide with an existing function (possibly created by someone else on the team). So what if the CLR executable you're about to deploy already exists? Well, in this case, you have some issues to deal with:

  • If the executable has been referenced in a stored procedure or other executable, then you need to consider what impact changing or updating the existing executable will have on the functionality of the existing code. You can use the SSMS Object Explorer to determine any dependencieswhich you might miss from a visual inspection. For example, if you right-click on an existing function, you can choose "Show dependencies" to expose the "Object Dependencies" dialog (shown in Figure 13.13).

    Figure 13.13. Viewing object dependencies using SQL Server Management Studio.

    In this case, the deployed function is dependent on a stored procedure TextCLRTempFunctions, which might break if you change or drop the function. Does this stop SQL Server from dropping the executable on demand? Nope. SQL Server ignores dependencies when removing functions and stored procedures.

  • If the executable is a UDT, then things get even uglier. In this case, dependencies are cast in stone as far as SQL Server is concernedyou won't be able to change a UDT once it's committed to a SQL Server table. This means your test scripts will likely create the test table using the new UDT, populate it with test data, dump its contents with a SELECT (or use it in a stored procedure), drop the table, and delete the UDT. I'll discuss easier ways to do this a bit later in this chapter.

Deploying a CLR Executable Function

If you think it's okay to deploy (your new function won't break existing functionality), you can proceed to deployment. Note that the default Solution Configuration automatically builds and deploys the CLR executable being developed when you click on the green debug arrow or press F5[6]. During the (build) deploy cycle, the Visual Studio Output (build) window displays messages returned from the compiler as Visual Studio builds and deploys your CLR executable. Note that several files in the assembly are posted to the server, where they're added to the Assemblies in the target database:

[6] Some keyboard layouts re-assign the function keys. Both the Visual Basic .NET and C# keyboard assignments use F5 to start debugging.

  • The compiled executable DLL, which contains the binary executable.

  • The project .PDB file, which contains the "program database" that holds debugging information and project state information used by SQL Server to step through the executable during interactive debugging. This file is altered to reflect the type of DLL being deployed (debug or production). Note that production builds cannot be interactively debugged.

  • The executable source code file(s). For Visual Basic .NET projects, these are the .vb files; for C# projects, these are the .cs files.

  • The assemblyinfo.(vb/cs) file, which names and describes the assembly. This file contains the assembly GUID, copyright, and version information. This file is generated via the project properties page.

It's always a good practice to check the Output Build window to see if the build and deploy phases completed before continuing. Yes, your application should deploy without issue unless there are frozen dependencies that block completion of the process. Of course, this step requires access to the designated SQL Server and sufficient rights to create the executables. Don't take this for granted. Most DBAs don't grant these rights to just anyone.

As you move back and forth from Visual Studio to SQL Server, you'll want to right-click and choose "Refresh" to ensure that just-added objects are visible in the Object and Server Explorers.

Profiling the Deployment Process

A tremendous amount of work goes on behind the scenes when you deploy your CLR executable. It's well beyond your "need to know," as far as I'm concerned, but if you're interested or have some issue that can't seem to be resolved by letting Visual Studio deploy your CLR executable, it might be helpful to understand what's going on behind the curtains when you click "Deploy" (as witnessed by the SQL Profiler):

  • The current CLR Assembly is dropped (if it exists).

  • The new CLR Assembly is created and the binary code is uploaded to the server.

  • The Assembly class's entry points are exposed as functions (or stored procedures or whatever they are).

Yes, you can use the Profiler to watch these operations flash by as you deploy your CLR executable, and you can learn enough about the process to replicate it using your own T-SQL code, if necessary. Some authors would have you think that using T-SQL is the only way to deploy CLR Assemblies, but they're not aware of what Visual Studio can door they don't care. The following steps walk you through the process of creating deploy scripts (leveraging an existing CLR Assembly) to deal with complex deployment issuesor to simply document what you've done. That's something that makes DBAs happy, and we all know how important that is.

1.
T-SQL requires that you drop any previous version of the Assembly before replacing it. Apparently, the ALTER ASSEMBLY does not permit overwriting an existing assembly. Using SQL Server Management Studio, right-click on a CLR assembly that's already been deployed and choose "Script Function" and "Drop to New Query Editor Window". This generates a T-SQL script, as shown in Figure 13.14, that drops the assembly. Don't execute this script just yet.

Figure 13.14. The T-SQL script to drop an existing assembly.

2.
Next, we need to create a new assembly. Using SQL Server Management Studio, right-click on a CLR assembly that's already been deployed and choose "Script Function" and "Create to New Query Editor Window". This generates a T-SQL script (shown in Figure 13.15) that creates a new ASSEMBLY.

Figure 13.15. Creating a new Assembly from the binary representations of the component files.

[View full size image]

Note that this script passes the binary representation of the component files. Of course, this would be kinda tough to do if all you have are the files themselves. An easier (and more practical approach) would be to generate the assembly from the component files, as done in Figure 13.16. Note that I moved the project to a shorter path for readability. I don't ordinarily save projects in the C:\ root directoryexcept on long weekends.

Figure 13.16. Creating a new assembly from the component files.

[View full size image]

The CREATE ASSEMBLY operator in T-SQL also permits me to indicate the level of trust I grant to the Assembly. I discuss these trust settings and the potential security risk they impose later in this chapter. Suffice it to say that a PERMISSION_SET SAFE project can do little or no harm. It means the code being executed is not at all trusted to keep out of trouble and cause damage. This is the default security setting.

3.
Next, you'll need to create the entry points for your DLL as functions in SQL Server. This can be done using SQL Server Management Studio by right-clicking on a CLR executable that's already been deployed. Choose "Script Function" and "Create to New Query Editor Window". This generates a T-SQL script (shown in Figure 13.17) that references the CLR executable source file (clsTempConversions.vb) and other configuration files (as previously discussed) already uploaded to the server in the CREATE ASSEMBLY step.

Figure 13.17. Scripting an existing CLR executable.

[View full size image]

No, as I said, in most cases, you don't need to go through all of this when deploying your CLR executable.


Setting Up a Test Environment

There are a number of ways to test your CLR executable using Visual Studio once it's deployed, but I don't expect you'll be that happy with the default behavior unless you make a few changes before trying to use it. Consider that SQL Server CLR executables are typically called from T-SQL stored procedures, functions, triggers, scripts, or batches. No, it does not make much sense to call a CLR-based stored procedure from a Windows forms application, ASP application, or Web Serviceyou could simply execute the same code from within the context of the application. So, my testing scenarios will be based on T-SQL scripts. This means if your skills lean toward SQL Server Management Studio (SSMS), go for it. The test scripts I created in the early days before Visual Studio debugging was hooked up were executed by SSMS. Now that Visual Studio debugging is wired, I'm still using SSMS. It still exposes more developer-friendly features.

When you try to debug a function with Visual Studio (by clicking the debug "run" button), Visual Studio looks for a default "debug script". However, before this T-SQL is executed, Visual Studio (by default) insists on rebuilding and redeploying the project. This can be problematic with CLR User-Defined types and some of the more complex CLR executables. To get around this problem, I created a special solution configuration used just for testingI'll discuss this a bit later.

To help choose a testing scenario that makes sense for you and best leverages your skills, let's look at a number of ways you can test your new SQL Server CLR executable:

  • Visual Studio includes a fairly simple (but unpopulated) SQL script (test.sql) that can be used to test your newly created functionassuming you add the T-SQL to invoke your newly defined executable. While the script shows how to call some of the more basic CLR executables, none of the specifics is filled inyou'll have to do that yourself.

  • Code your own T-SQL test script (as I'm about to illustrate) and add it to the project. Point Visual Studio at this script by setting the "default test script" (right-click the project).

  • Use SQL Server Management Studio's Object Explorer to generate a "SELECT TO" script that can be used to test the executable in a Query window.

  • Use a custom test script that's executed by the SQLCMD command-line tool or the SQL Server Management Studio Query window.

Addressing the CLR Function in T-SQL

CLR functions must be explicitly referenced in T-SQL by the owner. This means that you'll need to add the owner name to the function name to call in your test script, as shown in Figure 13.18. Yes, in this script, the database name is also includedit does not hurt to be over-explicit when addressing objects. It saves SQL Server some time when referencing the object.

Figure 13.18. Addressing a CLR function

Incidentally, this script was generated by SQL Server Management Studio. I simply right-clicked on the selected function and choose "Script to SELECT TO" in a new query window.

To use this script, you'll have to replace the parameter placeholder <@dblTempIn, float,> with a parameter value.

Notice that this (scalar) function's "return" value is captured by a SELECT. Remember, you can code the CLR function as you would any T-SQL function, as shown in Figure 13.19. In this case, I call the Fahrenheit-to-Celsius conversion function with input parameters of 32° and 212° (freezing and boiling, for those who skipped Physics 101).

Figure 13.19. Test script to exercise the fnFahrenheitToCelsius function.

[View full size image]

Executing a Custom Test Script

Now that I have a custom test script, I'm ready to test it with Visual Studio or SQL Server Management Studio. Let's step through the process of importing a new test script and executing it.

1.
Right-click the "Test Scripts" folder in your project and choose "Add Test Script". This opens a new test script. Paste the T-SQL from SQL Server Management Studio or T-SQL editor. An alternative to this approach is to choose "Add" | "Existing Item" and browse to the query script .SQL file you've previously built.

2.
Remember, I'm in Visual Studio now, so you can set breakpoints in this test script file. I'll set one on the first SET statement (line 5 in Figure 13.20).

Figure 13.20. Setting a breakpoint in the new SQL test script.

[View full size image]

3.
To get Visual Studio to use test1.sql as the default test script, right-click on the selected file and choose "Set as Default Debug Script".

4.
You're ready to step into this code, so press F5. Note that the build and deploy operations are repeated. While this is not that terrible when working with functions and stored procedure CLR executables, when you start working with UDTs, this approach simply won't work. They can't be rebuilt that easily.

5.
The test script stops at the breakpoint (line 5). Watch the Output (Debug) window as you step into the test script. The next line (line 6) executes the newly deployed function, which causes SQL Server to load a long set of DLLs to support the CLR executable. These include:

  • Mscorlib.DLL

  • SqlAccess.DLL

  • System.Data.DLL

  • System.DLL

  • System.Transactions.DLL

  • System.Security.DLL

  • System.XML.DLL

  • Microsoft.VisualBasic.DLL

  • The application symbols (when debugging)

Once the DLLs are loaded, Visual Studio executes an "auto-attach" to attach the debugger to the target SQL Server process. I show you how to attach to a specific SQL Server instance later in this chapter.

If you don't stop execution with a breakpoint, the text displayed in the Output (debug) window might not appear in sequential order. For example, without a breakpoint, the DLLs appear to be loaded and the auto-attach occurs after the SELECT executes.

The SELECT statements executed in the test script output their data to the Output (debug) window, as shown in Figure 13.21.

Figure 13.21. The Output (debug) window displays output from the test script.

[View full size image]

As you can see, creating a CLR executable function is not that hard. However, the question remains, is this more efficient than executing the equivalent logic in T-SQL?

Evaluating CLR Executable Performance

Let's take this simple (thus, easily understood) temperature-conversion function and build a test script that exercises the function and records the length of time it takes to perform it in a way that can be used to evaluate the performance. Ah, you need to keep in mind that when testing performance, you need to simulate the actual production configuration as closely as possible. No, you won't always be able to set up a multi-million-row test database, but one thing you'll need to do is avoid using Visual Studio test scripts to evaluate performance. As you'll see, Visual Studio adds considerable overhead while debugging any SQL executable, whether it's a stored procedure or CLR executableas long as you've enabled "debug" mode on the connection.

Tip

Debug mode with SQL Server debugging enabled can impact the behavior of the connection pool.

The test script used to evaluate the performance difference between calling the temperature conversion CLR executable function and the equivalent T-SQL is shown in Figure 13.22. This script was run on SQL Server Management Studio and again as a test script in Visual Studio. The difference was dramatic. In Visual Studio, the script ran in debug mode took 10 times longer to execute the SQL CLR code. This is due (in part) to the extra overhead Visual Studio imposes when you ask to debug the CLR procedure using the designated test script.

Figure 13.22. Test script to compare CLR executable vs. T-SQL execution times.

[View full size image]

This means you'll want to build test scripts to test and debug (and step through) CLR executables using Visual Studio, but that's it. Don't depend on the timings you get back from Visual Studio for performance metrics. I created a report to show the results of these test runs as shown in Figure 13.23. Note the scale of the X-axis.

Figure 13.23. Comparing CLR executable and T-SQL performance.

[View full size image]

In this test (using SQL Server Management Studio), the CLR-based temperature conversion function runs over seven times slower than the equivalent T-SQL code. This means that the relatively simple task of performing a temperature conversion should not be done in a CLR executableyes, this is an example of what not to do.

Let's take a look at a few situations where CLR executables are supposed to make more sense.

IMHO

Are CLR executables a solution looking for a problem?

Evaluating the GPS Example

I keep hearing that the best use of CLR executables is where T-SQL is forced to perform "CPU-intensive" or complex mathematical functions. I've also heard that the calculations needed to work with Global Positioning Satellite (GPS) data are sufficiently complex and that they could best be done in a CLR executable. To this end, I created a sample database[7] that has the (rough) GPS coordinates of about 1,200 U.S. cities. This data was fed into a formula to determine the distance between each of these cities. The resulting table has about 1.5 million rows. The distance calculation is performed in both a T-SQL function and an equivalent CLR functionthe test suite compares the total execution time of both techniques. In this test, the CLR implementation is slower, but only by .005% when processing the 1,478,523 rowsbasically, a dead heat. However, in this case, I found it easier to code the CLR function, as it affords a far richer choice in conditional logic and error checking, as well as a familiar development toolset. I expect T-SQL developers might find just the opposite experience. I also expect that as the computational needs of your project increase, the difference in performance will be more pronouncedthat is, the CLR implementation might make more sense.

[7] This is the GPS database on the book's DVD.

The bottom line? Well, given that the CLR GPS calculation came pretty close to matching T-SQL performance, there are going to be situations where CLR executables can make a (positive) performance difference. I would expect these "better" solutions to be more complex than the single GPS distance calculation before you could justify their use on the basis of performance alone. Let's keep experimenting to see if there are other ways to use a CLR executable that can really justify their use.

Creating a Table Value Function

One of the more interesting (and useful) features first enabled in SQL Server 2000 is Table-valued functions (TVF), which permit T-SQL functions to return a code-generated rowset to other T-SQL logic, such as a SELECT statement. I talked about this functionality before when I showed you how to implement a T-SQL IN clause that takes a dynamic list of arguments passed as a delimited string. In this example, I used a T-SQL TVF to convert a delimited string to a SQL Server Table that's fed into the IN clause via a SELECT. Based on the number of times developers ask for this functionality, I'm surprised that Microsoft has not included this functionality in T-SQL itself.

At this point, I want to show you how to implement a TVF using a CLR function. It's only a bit more involved than a typical CLR function, as you have to implement the interfaces that the CLR compiler and SQL Server expect to see when you return a "table" structure. An important difference between T-SQL TVFs and those you're about to build in a CLR function is that a T-SQL TVF is persisted (at least temporarily) to TempDb before the results are returned to the caller. This means disk I/O is required to implement them. In the CLR case, the rows are streamed into memory instead of stored to disk[8], so the rows can be passed immediately to the caller as they're constructed.

[8] I expect that the data "inserted" into a T-SQL TVF is actually cached to memorynot hard disk, at least for smaller tables.

Coding the TVF

In order to implement a CLR streaming TVF, you'll have to add support for the IEnumerable Framework interface. This is really pretty easy (despite what you might have read in some of the online articles). You'll also need to set specific attributes on the CLR function so that the complier knows what the function is trying to return.

Start a new project and set up a CLR function as I did in the previous exercise. In this case, I need to tell the compiler that I plan to return a table. This is accomplished by defining the FillRowMethodName and TableDefinition in addition to the function Name. To help remind you of these argument attributes on the SqlFunction definition, Visual Studio exposes an Intellisense dialog, as shown in Figure 13.24.

Figure 13.24. Visual Studio helps define the function attributes with Intellisense.

[View full size image]

In this case, I'm going to call the TVF "DelimitedStringToTable" and name the routine that returns "items" from the generated table by a routine named "GetItems". The table returned is defined with a single nvarchar column that limits the individual delimited items to 500 (Unicode) characters. That should be plenty. All of the work is done in the InitMethod routine that simply uses the Visual Basic (or C#) Split function to break up the delimited string into individual elements. Note that InitMethod is exposed as Ienumerable, which deals with all of the plumbing behind the scenes. The entire CLR function is shown in Figure 13.25.

Figure 13.25. The DelimitedStringToTable CLR function.

[View full size image]

I also wrote a T-SQL routine that duplicates the CLR Split function, shown in Figure 13.27, so I could compare performance between the two implementations. To test the two TVF routines, I created a test harness, as shown in Figure 13.26.

Figure 13.26. Testing the CLR and T-SQL TVF functions.

[View full size image]

In stark contrast to earlier tests, when I ran this test, I discovered that the T-SQL version is dramatically slower than the CLR version (by over three times). The fact that CLR languages know how to strip a delimited string with a single line of code probably plays an important role in the performance boost.

Figure 13.27. The DelimitedStringToTableTSQL T-SQL function implementation.

Evaluating Performance with the SQL Profiler

The difference in performance was measured by watching both queries execute with the SQL Profilerthe results are shown in Figure 13.28. If you inspect the Profiler trace, you can see that the CLR function test consumed 391 CPU milliseconds, while the T-SQL version took 1,219a big difference. But also note that the CLR function required no disk-I/O, while the T-SQL version required 25,000 I/O reads. Remember, a T-SQL TVF uses TempDB to store its data. It's also important to note that the duration was skewed to the CLR sidebut not as much as the difference in CPU time.

Figure 13.28. Using the Profiler to compare CLR and T-SQL TVF performance.

[View full size image]

It looks like we've found a good use for CLR executables. Yes, I agree that when you need to perform string manipulation with T-SQL (as when parsing XML), the Framework's native string-handling capabilities are far richer than those supported by T-SQL. This can help improve server-side executable performance.

Let's see how a TVF can help in an ordinary query. One of the ways to use TVFs is to process pick lists. For example, when using a multi-select ListBox used to pick one or more items to focus the WHERE clause, the T-SQL can use an IN expression to narrow the resultset. One of the problems with the IN expression is that it can't accept a parameter. However, it can accept a rowset from a SELECT. Here's where I use the CLR TVF (DelimitedStringToTable) to feed the IN expression. The example in Figure 13.29 shows how this is implemented in a stored procedure.

Figure 13.29. Using a CLR Table Value Function to feed an IN expression.

[View full size image]

And Figure 13.30 shows how I tested this stored procedure by passing a delimited string as a stored procedure parameter.

Figure 13.30. Passing a delimited string to an IN clause via a TVF.

Previous Page
Next Page


Working with ADO.NET from Within CLR Executables

Before wading into a hip-deep discussion of CLR stored procedures, we need to take a short side trip to acquaint you with ADO.NET and how it's used (or abused) in a CLR executable. Yes, there are times when your CLR Executable code needs to access data from the local SQL Server or perhaps from an external source like another SQL Server instance or data in another format, in another database, or in Cleveland. The engineers at Microsoft experimented with this interface for quite some time during the beta cycle, and we all saw a number of implementations come and go. You'll see evidence of these tried-and-abandoned approaches in a number of magazine articles and stale Google content.

Using the Context Connection

To simplify your job, when it comes time to connect to a data source, you have two fundamental choices, as illustrated in Figure 13.31:

  • Use the SqlContext connection. In this case, the CLR executable leverages the fact that the code is being executed in the process space of a local SQL Server instance. If this is where your data is located, it makes abundant sense to use this approach to connect to SQL Server. This approach bypasses the entire protocol and transport layers because, essentially, SQL Server is talking to itself to get the datajust as it does when it executes T-SQL queries.

  • Use a "normal" ADO.NET SqlClient connection. If you need to access data from an external source, you can choose to open a SqlConnection that routes your connection out through the protocol and transport, and back in through the matching protocol layers on the target server. This is the same technique used by other clients accessing SQL Server over the network or from the same system[9].

    [9] Yes, you can connect to SQL Server using the shared memory provider if the executable is running on the same system as SQL Server. This interface bypasses the transport layers.

Figure 13.31. Connecting to SQL Server via a "context" or "normal" connection.

[View full size image]

A context connection is created when you specify "context connection=true" in the ConnectionStringyou won't be permitted to add any other keywords. Just a few caveats when using a context connection:

  • You can open only one context connection. Yes, multiple instances of your executable will each get their own context connectionin their own context.

  • No, you can't use MARS with a context connection, nor can you use SqlBulkCopy, update batching (as supported in ADO.NET 2.0), or SqlNotificationRequest.

  • You also cannot cancel an operation in a context connection. SqlCommand.Cancel() is ignored.

There are also a couple of restrictions on using a "normal" connection from within a CLR stored procedureno asynchronous operations, and the SqlDependency object is not supported. You'll also want to stay away from Integrated Security (SSPI) authentication, as the "identity" used for the connection will be the identity of the SQL Server servicenot yours. I'll show some examples of using a context connection later in this chapterseveral of the CLR executables use it.

Sure, you can use a "normal" connection to connect to other data sourceseven Oracle and JET. Yes, these connections will limit the performance of your SQL Server, as it has to wait for the source database engine to be instantiated, accessed, and queried.

When you access an external database, you'll have to mark your CLR executable Permission Level to "EXTERNAL_ACCESS". The implications of this setting are discussed later in this chapterit's set via the Project Properties, shown in Figure 13.32. A context connection can run with the default setting"Safe".

Figure 13.32. Setting the Permission Level with the Project Settings dialog.

[View full size image]

Managing Transactions

CLR Executables have been integrated with the 2.0 Framework's System.Transactions. You'll have to manually add a reference to System.Transactions.DLL in your CLR executable References list to get access to this functionality in your code. To get a better idea of how the CLR works with SQL Server to implement transaction support, I suggest reading Pablo Castro's article[10] "Managed Data Access Inside SQL Server with ADO.NET and SQLCLR," posted on MSDN. Generally, it's recommended that you create your own transaction scope within the context of your CLR executable. The CLR and SQL Server work together to coordinate your transaction scope with the transaction context used to execute the executable.

[10] http://msdn.microsoft.com/sql/default.aspx?pull=/library/en-us/dnsql90/html/mandataaccess.asp#mandataac_topic10

Querying Data

Once the ADO.NET Connection is open (yes, you still have to open and close the Connectioneven a context connection), you use it as you would any normal connection to SQL Server. You still use the SqlClient namespace classes to create SqlCommand instances, populate their Parameters collection, and execute them. You can create SqlDataAdapter instances or simply create a SqlDataReader to return single or multiple resultsets. The next section has plenty of examples of executing SqlCommand objects and working with the resultsets.

Returning Data and Messages

Unlike a "normal" Windows application, a CLR stored procedure or other executable often has to return special structures like Tables (in a Table-value function), one or more resultsets (that might contain one or more rowsets), messages, and exceptions. The following example illustrates how to use the SqlContext.Pipe to create these resultsets that are sent directly back to SQL Server. All data returned is sent to the local SQL Server instance hosting the CLR executable through this context pipe.

When implementing a CLR User-Defined type (UDT) or aggregate, there are special interfaces used to expose the data you build to SQL Server. These are also explained in great detail later in this chapter.


Coding CLR Stored Procedures

Now that you understand how to build CLR functions, you're ready to step through the implementation of the other CLR executables. Thankfully, most of the other executable types share a few attributes in common, so you won't have to start from scratch. We start with stored procedureslike CLR functions, CLR stored procedures can be prototyped and tested quite easily with Visual Studio 2005. However, there are a number of tips I can provide to make the process of coding, deploying, and debugging your CLR stored procedure easier. Let's start by comparing functions and stored procedures. Table 13.1 highlights the fundamental differences.

Table 13.1. Functions Versus Stored Procedures

Function

Stored Procedure

Supports 0 to N input parameters

Supports 0 to N input parameters

No OUTPUT parameters

Can return N OUTPUT parameters

Returns a single value

Returns a SqlInt32

Can return a T-SQL table

Can return N Rowsets

Cannot return messages

Can return "PRINT" or "RAISERROR" messages

Note that functions are generally used in expressions and aren't meant to return more than a single rowset (as a SQL Table). The function's return value is more flexible, in that you can return almost any datatype, while a stored procedure RETURN returns only a SqlInt32 value. However, that's deceptive, in that a stored procedure can return virtually any number of OUTPUT parameters of almost any type. When stored procedures are coded, the arguments passed in the "parameters collection" (the arguments passed to the CLR stored procedure) must be specifically designated so the CLR compiler knows which parameters are input and which are output. Sure, an output parameter can also accept an input value, and in testing, you might have to force a value to get Visual Studio to accept it. Later, I'll show you how to return one or more rowsets and "PRINT" messages from a CLR stored procedureagain, something you can't return from a function.

Tip

Later in this chapter, I illustrate how to code and debug a CLR stored procedure that accepts a CLR UDT as an input parameter.

The example stored procedure I'm going to build addresses one of the reputed weaknesses in T-SQLnamely, string handling. The challenge is to search a varchar(max)[11] column for a specific set of words and report the hit count. The CHARINDEX function in T-SQL does a pretty good job of this, but it's only part of the coded needed to manage the incoming list of search words and the resulting output rowset.

[11] varchar(max), nvarchar(max) are designed to supplant the TEXT datatype.

IMHO

When building any CLR executable, you have to balance the performance cost or benefit with the amount of developer productivity you gain or lose.

In the process of building this example, I created four versions of the codeeach a bit more sophisticated than the last. This approach should help you get a better understanding of the issues one at a time and not overwhelm you with a completed (albeit somewhat complex) end-product. Each of these examples is driven from a delimited list of words to locate in the BookContent column of the BookText table in the BookSource database (created just for this chapter). These examples[12] can be briefly described as:

[12] All of these examples are on the DVD under "CLR Stored Procedures."

  • The first version implemented the code using a T-SQL stored procedure (CountWords) that calls the aforementioned delimited stringtotable function and another T-SQL function to count the words in a string. The stored procedure plus the T-SQL functions total about 100 lines of T-SQL, including the comments. This is the benchmark against which the CLR procedures will be judged. The code for this stored procedure and companion functions are all on the DVD.

  • The remaining examples are all built using CLR stored procedures. The first of these CLR implementations (V1) closely approximates the approach used by the T-SQL example, which approximates what a developer might implement when converting an existing T-SQL routine. It executes the CountWordsInString function in conjunction with a SELECT statement that's executed for each word in the list of words to locate. It uses a Structure (C# struct) collection to store the data in memory. The backend of this example spools the contents of the in-memory collection back to SQL Server as a rowset. I'll show you exactly how that's done later in this section.

  • The second CLR example (V2) saves the results of the initial SELECT to a DataTable that's populated using a DataReader. This was implemented to permit easy sorting of the output rowset (as implemented in the T-SQL routine). I had planned to use a high-speed dump of the DataTable back to SQL Server using an extruded DataReader, but as I describe later in this section, the SqlDataReader required by the SqlPipe interface was not compatible with the DataTableReader returned by the new 2.0 function.

  • The third CLR example (V3) took a radical approachit did not use the server-side function to locate the words in the selected strings. Instead, I rolled my own FindWordInString function within the CLR stored procedure. This simple-looking routine takes advantage of the new IndexOf function exposed by the 2.0 Framework. It took considerable time to debug this routine so it matched the functionality of the T-SQL implementation. The results were pretty spectacular as I'll show you when we get to the performance comparisons later in this section.

Building the Example CLR Stored Procedures

Creating a CLR stored procedure using Visual Studio is not much different than creating a CLR function, so I'll skip down to the stuff that is different. Thankfully, testing is usually pretty easy as welluntil Visual Studio decides that it's quitting time when it seems that nothing will get it to work right. I'll show you how to avoid those situations and when to just reboot and start over. Again, it's always a good idea to keep saving your applications and exit Visual Studio from time to time to let it clean its house of leftover application chad.

Let's walk through the process of creating your first example CLR stored procedure.

1.
Start a new SQL Server Database project and add a new itemin this case, a Stored Procedure.

This opens the default stored procedure template (as shown in Figure 13.33), which is not completely correct, so it's not that helpful. I say that because the template does not hint about how to pass input or output parameters or set the RETURN value. The beta versions I saw had some of this functionality included in the template. That's okay; I can add it back fairly easily. Let's step through the process of adding these features to my stored procedure.

Figure 13.33. The unmolested CLR stored procedure template.

2.
Because the template suggests that you should use a Sub for a stored procedure, you're going to need to make some fundamental changes. A stored procedure (like a function) returns a value set by the RETURN statement in a T-SQL stored procedure or the Return operator in Visual Basic .NET. Yes, in the case of a stored procedure, the return value always cast as a SqlInt32. I generally use the RETURN value to set a "success" value0 means the stored procedure worked, and something else like 1 means it didn't.

So, I really need to implement a standard stored procedure. The first thing I must do is change the stored procedure CLR "subroutine" code to a function and add the AS SqlInt32 clause to the Function declaration, as shown in Figure 13.34. Sure, it would be a great idea to change the Visual Studio template as well so that each new CLR stored procedure is created correctly.

Figure 13.34. The "function" version of the CLR stored procedure implementation.

[View full size image]

The Visual Basic language does not verify that the Return and output parameters are set in your code. This is something that the C# compiler flags as warnings when you build the project. This helps ensure that all of these values are set before exiting the routine.

Note that in Figure 13.34 (and the C# version in Figure 13.35), I've filled in the three parameters to be passed to and from the stored procedure. New for the RTM version[13] is the requirement that output parameters be marked with the "<Out()>" attribute in addition to being passed ByRef (by reference) instead of ByVal (by value, the default). The <Out()> attribute is not needed in C#, as the language supports parameter direction attributesthe C# version of the template is shown in Figure 13.35. Note that Microsoft incorrectly assumes that you need a "subroutine" instead of a "function" template. The fleshed-out code illustrates that you must set the output and return value before exiting the routine.

[13] I worked with the "alpha", beta, and CTP versions of SQL Server and Visual Studio for about two years before seeing the final shipping version. It changed a lot over the years.

Figure 13.35. The C# prototype uses somewhat different notation.

[View full size image]

3.
The first task the example takes on is to convert the input parameter containing the delimited list of words to use as search arguments into an array. This is accomplished in a single line of code, as shown in Figure 13.36. Remember, this took about 20 lines of T-SQL to implement. At this point, you're already ready to set the first OUTPUT parameter. This is easy; just assign a value to the variable declared in the stored procedure function declaration header.

Figure 13.36. Parsing the delimited list into an array of search words.

The next job the CLR stored procedure has to accomplish is to fetch the string (varchar(max)) from the BookSource database' BookText table. This is done by creating and opening an ADO.NET Connection instance. The ConnectionString property, in this case, is specialunique to CLR stored procedures. While made much simpler than the early betas, this concept sometimes confuses developers. Consider that this stored procedure is being executed on a specific SQL Server instance, so there is no need to open a connection to the serveryou're already "connected." Yes, you do need to tell ADO.NET that the query you're about to execute is to be run on the current context's SQL Server instance by setting the ConnectionString to "context connection=true", as shown in Figure 13.37 (line 24). No, there is no connection pool involved here, but it's still a good idea to close this Connection when you're done.

Figure 13.37. Opening a Connection to the current SQL Server instance.

[View full size image]

4.
Until I'm ready to return the results from the stored procedure I can proceed just as if I have set up a remote Connection. I'm ready to set up the query using a SqlCommand and populate the Parameters collection. Figure 13.38 illustrates how this is done. It should look pretty familiar by nowit's just an ordinary SqlCommand object instantiation.

Figure 13.38. Creating a simple Command object.

[View full size image]

5.
Your next task is to execute the SqlCommand once for each word in the list. Yes, it might be more efficient to avoid all of these queries, but these "round-trips" are far cheaper than those executed from a networked client. Note that the code still has exception handlers, but you can't include any "debug.print" messages or expose a dialog to the user to notify him or her of an issuethere is no "user". This code is running on a server somewhere, and even if the CLR would let you expose a dialog, there's no display to show it on. It's the task of the code calling this stored procedure to deal with these exceptions.

Figure 13.39. Execute the query for each word in the collection.

[View full size image]

6.
Once the SqlDataReader returns rows, it's time to capture the data returned. Remember, this rowset has the string used as a search argument (the "word"), the hitcount, and the name of the original source file. As I said earlier, this code saves this rowset data to a collection, as shown in Figure 13.40. Yes, you'll have to close the DataReader between invocations of the ExecuteReader methodjust like in a client/server or ASP.NET application. Note that I'm not performing anything time-consuming in this routineI don't want to delay rowset population any longer than necessary, so as to free the locks as soon as possible.

Figure 13.40. Persisting the returned rowset in a Collection.

7.
Once the SqlDataReader returns False from the Read, you know that there is no more data to process. You can close the connection. You're now ready to proceed to phase 2returning the results of the query as one or more resultsets.

Using the SqlContext Class to Return Resultsets

New for the 2.0 Framework, the SqlContext class is used to expose properties and methods that are passed directly to the local SQL Server instance hosting the CLR code you're executingthe local "context". This context is used whenever working with stored procedures or any other CLR code that needs to access the host server. For example, when implementing a CLR trigger (which is simply a form of stored procedure), the SqlContext exposes the pseudo "inserted" and "deleted" tablesjust like the pseudo tables exposed to T-SQL triggers.

Let's take a short tour of the SqlContext class and highlight the most interesting points of interest:

  • The IsAvailable property is useful to determine if the currently executing code is indeed running in processin context with a SQL Server instance. Unless IsAvailable is True, you won't be able to use any of the other SqlContext objects, properties, or methods.

  • The SqlPipe object is basically a pathway back to the local SQL Server context. You write resultsets to the pipe, as illustrated later in this section.

  • The SqlPipe supports several methods that can send strings, which appear in the same way as PRINT messages. You can also use other methods on the SqlPipe object to send single-row or multi-row resultsets, or execute a query and return the rowset. You can also create a SqlDataReader and return the data stream directly to SQL Server via the SqlPipe object. I'll show examples of each of these techniques later in this section.

  • The SqlPipe also permits you to as many multi-row resultsets as needed between calls to the SendResultsStart and SendResultsEnd methods. The data returned is formatted using the SqlDataRecord class to lay out the schema.

  • The SqlTriggerContext class describes the context in which triggers are invoked, including an XML EventData structure that describes the operation that fired the trigger. I illustrate how to code a CLR Trigger later in this chapter.

I use the SqlContext class to send the example's persisted rowset back to SQL Server as a resultset. Note that you don't need to have an open Connection for the SqlContext properties and methods to work. An open Connection (pointing to the local context) is required only when using ADO.NET to execute a SqlCommand in the local context. Let's continue stepping through phase 2 of the example.

1.
Start by signaling the client calling this stored procedure that Phase 1 of the stored procedure succeeded. This isn't really necessary, but it's a simple way to illustrate use of informational messages to report progress or other notable events. The code to send this message is shown in Figure 13.41.

Figure 13.41. Returning a PRINT message to the client using the SqlContext object.

[View full size image]

2.
Next, you need to return a multi-row resultset to the client[14] containing the results from the earlier string-search algorithm, so you'll need to describe the columns (fields) of the row to be returned. This is easysimply build an array of SqlMetaData objects and populate the array using the New constructor that sets the appropriate properties. Then, create a new SqlDataRecord object using this array of SqlMetaData objects. All of this is illustrated in Figure 13.42.

[14] The "client," in this case, is the routine calling the CLR stored procedure.

Figure 13.42. Defining the resultset schema using the SqlMetaData and SqlDataRecord classes.

[View full size image]

3.
Since you're sending a whole set of rows (and not just one), you need to set up the SqlContext Pipe, so it knows that it should bundle all of the rows together in a single resultset. This is accomplished by calling the SqlContext.Pipe.SendResultsStart method. Once the Pipe is "primed" for multiple rows, you must flush it by calling the SqlContext.Pipe.SendResultsEnd method, but before doing that, you need to send each data-populated row down the pipe using the SqlContext.Pipe.SendResultsRow method. All of this is illustrated in Figure 13.43.

Figure 13.43. Passing individual rows into the SqlContext.Pipe.

[View full size image]

When declaring variables using SqlTypes, be sure to initialize them as the default to NULLnot 0, as some might expect. Remember, X + NULL = NULL.

4.
Remember that I had a couple of OUTPUT parameters defined for this stored procedure as well as a RETURN value? These parameter values are set by simply assigning values to the variables defined in the stored procedure declaration header (as shown in Figure 13.44).

Figure 13.44. Setting the final OUTPUT and RETURN value parameters.

Basically, that's it for the CLR stored procedure code. Testing the procedure is our next challenge.

Testing the CLR Stored Procedure

You'll need a test script to try out the new CLR stored procedure. Visual Studio has created one of these for you (sort of)at least, it's given you a reasonable starting point to build a test script. Figure 13.45 is my test.sql script. This script should be installed in the projectassuming you want to debug the CLR stored procedure with Visual Studio. Nope, don't plan to evaluate performance using this techniqueas discussed earlier, Visual Studio adds a ton of overhead to the process of executing and stepping through the CLR code.

Figure 13.45. The test.sql test script for V1 of my CLR stored procedure.

Since I plan to debug using Visual Studio, this test script should be installed in the project using the Project Explorer. Yes, I discussed this earlier when building my first CLR function, so refer back there for more details on setting up a debug script if you skipped that section.

When you start the debug process (as described earlier), Visual Studio starts the test script. If you didn't set a breakpoint, the entire test script is executed and the resultsets are displayed in the Visual Studio "Output" (debug) window. The results are not formatted very neatly, but you can tell if the stored procedure threw an exception or returned the right set of data.

As you step through the code and reach the ExecuteReader code (line 38), Visual Studio displays a new window that contains the T-SQL about to be executed, along with the value of any parameter supplied. This feature is new for the RTM versionI can't recall seeing it in the Beta or CTP versions...but I could be wrong. Anyway, this is a handy way to determine just exactly what's getting executed. You can also inspect the contents of the input parameter using the new Visual Studio Intellisense feature (as shown in Figure 13.46).

Figure 13.46. The T-SQL being executed by the ExecuteReader method.

[View full size image]

You'll also discover that as you debug your CLR executable, Visual Studio walks you right into the T-SQL functions invoked by the T-SQL code being executed. In this case, you're taken into the source code for the CountWordsInString function; again, you can use Intellisense to see the contents of the T-SQL variablesjust select the variable. No, this version does not support edit and continue. That's a bit of a tougher problem to crack when the code is running on a remote SQL Server. I don't expect that I'll see that feature before we're out of Iraq.

Figure 13.47. Debugging with Visual Studio walks you into the underlying functions.

[View full size image]

Another interesting point that should be mentioned is how SQL Server manages the execution thread of this code. When debugging this routine (well before it was working), I found that stepping into the routines that worked with the resultset returned by the SqlDataReader was interrupted by jumps back into the CountWordsInString T-SQL function. That's because as SQL Server processed the query, it was running separate threads to manage both the "server side" of the query (the function invocation and row-fetch operations) and the "client side" (the CLR stored procedure code). This can be a bit confusing at first and might require that you trap exceptions and data value snapshots as you go.

Exploring the Remaining CountWords Examples

The second (V2) "count words" example is very similar to the first versionit simply uses a DataTable to persist the intermediate data instead of using a collection. I took this route because I wanted to illustrate how to use the DataTable object's DataView to sort the rowset before passing it back to the client. As I said before, I also wanted to illustrate the use of filling the SqlPipe with the results from a DataReader. Unfortunately, this approach failed, as the DataTable class's CreateDataReader does not extrude a data reader consumable by the SqlPipe method. This means that you can pass only a SqlDataReader to the SqlPipe object.

The third (V3) example is worth exploring a bit, as I took an entirely different approach to handling the job of locating words in the target column strings (varchar(max)). Instead of calling a T-SQL routine to find the words (as was done in the baseline T-SQL example), V3 takes on that task itself. The query used to return data to the CLR stored procedure is shown in Figure 13.48.

Figure 13.48. Returning the varchar(max) data to process.

The rowset returned from this query would overwhelm the client memory space if I created a DataTable to hold it, but I'm only going to work with a single varchar(max) data column at a timeas quickly as I can. No, this data is not being moved over the Net, so this approach should be dramatically faster than a client/server program that does the same job. You should also save time and CPU cycles by implementing the "search for words in a string" code locally instead of having SQL Server call a T-SQL routine. Sure, one alternative approach would have been to replace the T-SQL CountWordsInString function with an equivalent CLR function. Just to get an idea of what's going on behind the scenes, let's compare the two approaches. First, the T-SQL CountWordsInString function (shown in Figure 13.49).

Figure 13.49. The T-SQL CountWordsInString function.

The equivalent CLR CountWordsInString function (called as I out-process the rowset) is shown in Figure 13.50. Note that this routine would have been harder in earlier versions of .NET, as the IndexOf method was introduced in the 2.0 Framework. It's a pretty powerful method, as it permits you to choose a case-sensitive or non-case-sensitive search (along with other options).

Figure 13.50. The CLR CountWordsInString function.

Evaluating CLR vs. T-SQL Performance

As, with all of these versions, which do you expect to run faster than the T-SQL code? Given the pitiful performance of the temperature-conversion example, perhaps none of them. To test the routines, I set up a number of individual test scripts to run that include dumps of the operational statistics. You'll see what I mean in a minute. Let's take a look at one of the test scriptsthe one used to test and evaluate the T-SQL version. Remember, you can't use Visual Studio to test these routinesit adds too much overhead. The base test script is shown in Figure 13.51. Each example can be called from this templateI simply change the name of the stored procedure to execute.

Figure 13.51. Test script to exercise the CountWords examples.

To fairly measure the four examples, I ran this same script using SQL Server Management Studio with the "Show Client Statistics" option enabled, which is exposed as a button on the toolbar shared by the Execute button (as shown in Figure 13.52).

Figure 13.52. Enabling client statistics diagram in SQL Server Management Studio.

Once enabled, SQL Server Management Studio captures and persists operational statistics about each query executed in this window. By changing the name of the stored procedure executed for each test, you can to get a rough idea of the overall performance among the various implementations. I ran each version once, which might favor the first or last routine executed because of caching, so I re-ran the T-SQL example again to check the validity of the resultsthey were identical. Sure, I had run the test a dozen times beforehand, so the data was probably all sitting in the data cache, so disk I/O would not be a factor in this evaluation.

The results from all four tests (T-SQL and the three CLR implementations) are shown in Figure 13.53.

Figure 13.53. The client statistics chart for the CountWords examples.

[View full size image]

If you look closely, you can see that the bottom-line results are pretty startling. Microsoft was actually rightthe CLR code is faster, a lot faster. The final CLR example shown as Trial 4 in the chart took only 78ms to execute, while the T-SQL version took 640msover eight times faster. Note that the other CLR implementations did not do any better (but not really any worse) until I incorporated my own CountWordsInString function in the CLR stored procedure.


Implementing Basic User-Defined Type (UDT) CLR Executables

One of the more appealing features of CLR executables is the ability to define custom user-defined datatypes, or UDTs, for short. Like the sirens of Homer's Odyssey, CLR UDTs can be a very attractive way to gain functionality, but at the same time, it can be hard to prevent your development team from sailing up on the rocks. No, I don't recommend putting wax in their ears. While it worked for Odysseus, his crew was too deaf to avoid the Charybdis and Scylla later in the voyage. The challenges of choosing to implement CLR UDTs include:

  • CLR UDTs can add significant complexity to your database and the applications, tools, and training needed to access and manage them.

  • Once a table is defined with a CLR UDT, you won't be able to change the UDT without dropping the table column. The current technology does not support any "versioning," so you'll have to perform all data transmogrifications yourself. However, as I'll illustrate, it's possible to create a UDT that can accept data in a number of formats. This means that you'll be able to more easily migrate from one implementation to the next.

  • While a UDT can be referenced in code not hosted in the SQL Server database, it requires another set of steps. This involves manually adding a reference to the host project and understanding how to reference, format input data, and convert from one format to another. All of these points are illustrated in this section, as well as in subsequent examples later in the chapter, where I use the UDT in an aggregate, in a Windows form, and in a CLR trigger.

  • Referencing UDT public properties exposed by the CLR code can be problematic, as these are hidden from developers that don't have the source code for the UDT. Without the source code, you won't be able to use Visual Studio or SQL Server Management Studio to explore a UDT. Yes, I expect that a UDT can be explored with a decompiler like Reflector.

  • You'll discover that you can't SET more than one UDT property at a time in an UPDATE or INSERT statement. This means you'll need to write custom code to make changes to rows containing UDTs.

  • Some UDT implementations prohibit use of SELECT * to return all rows and all columns because of the way that NULL is handled. Yes, the SQL Server Management Studio and Visual Studio tools bypass this by simply referencing the columns by name in the SELECT.

  • The Visual Studio 2005 IDE designers and wizards, like the Data Source or TableAdapter Configuration wizards, don't support UDTs. This means you won't be able to auto-generate strongly typed DataSet classes against tables that contain a UDT. However, you can create a VIEW on a table that contains a UDT and exposes the properties individuallythese can be used to generate read-only strongly typed TableAdapter objects.

  • You'll likely invoke the CLR code each time the UDT column is accessed. This means that the CLR UDT can add a significant performance penalty to your application, depending on its implementation and the complexity.

Okay, given these challenges, why would one want to bother creating a CLR User-Defined type? Well, suppose you have a data structure that lends itself to being contained in a single column. Examples that come immediately to mind include:

  • A "currency" datatype: In this case, you could store a currency value along with the monetary unit associated with the type. That is, you could store 234.56 currency value along with other properties to indicate the value is in British pounds. To implement this UDT, you could store the value using a high-precision monetary datatype, a currency symbol (£, in this case), the currency type (GBP), and the currency name ("British Pounds"). Once this UDT is implemented, you can aggregate (sum) columns regardless of the currency type by converting to a base type before adding the value to the sum. A currency UDT would also permit you to expose your data anywhere without fear of a localized value setting misinterpreting the intended value. Yes, this could be implemented with separate columns, but this assumes that the applications written to manage these values carefully tracked the currency type. Later in this chapter, I implement a currency UDT described here.

  • A "point" datatype: Many of the examples Microsoft uses in the documentation illustrate UDTs with a data structure that manages the x/y coordinates of a point. Since these values should be handled as an atomic unit, this is an ideal (albeit expensive) way to manage these values.

  • A "GPS" or latitude/longitude datatype: When referring to a GPS location in latitude and longitude, you're really dealing with eight values: latitude and longitude in hours, minutes, seconds, and declination. For example, 47° 40' 10" N 122° 3' 26" W is the GPS location for Redmond, Washington. Incidentally, this "degrees, minutes, and seconds" notation is referred to as "Sexagesimal," in case they ask you on Jeopardy. Again, GPS data is an ideal candidate to incorporate into a single atomic unit of information. The individual minutes, degrees, and seconds of latitude and longitude don't have much meaning in of themselvesit's only when they are used to form a complete address that they make sense for the data row to locate a specific point on the earth. This UDT is also implemented and discussed later in this chapter.

So, yes, there are specific cases where CLR UDTs seem to make sense. If you consider that the UDT implementation can embody comprehensive validation logic to ensure that each component meets specific range criteria, you can see how a CLR UDT can help keep data validregardless of the source.

Coding CLR UDTs

As with the other CLR executables I've created so far, you start building a CLR UDT using the same Visual Studio "Database" project type as previously discussed, but in this case, you should choose "User-Defined type" from the "Add New Items" menu. Name your UDT as neededin this case, I call it "typGPS". Our UDT is going to store the latitude and longitude for GPS data, as discussed earlier.

IMHO

I don't really expect this CLR UDT to improve overall performance.

Once you choose the UDT template, Visual Studio generates a block of code that exposes prototypes for all of the required properties and functions, as well as a couple of optional methods that you might choose to implement. This generated template (in compressed form) is shown in Figure 13.54. As you can see, a UDT is defined as a Visual Basic .NET Structure or C# struct that implements INullable.

Figure 13.54. The default CLR UDT template.

[View full size image]

Along with the critical functions and properties, the CLR UDT prototypes a pair of methods that you can flesh out if your UDT needs them. For example, if you need to call an initializer, you might want to implement it in one of these methods.

Defining the CLR Public Properties

When it comes time to use your UDT in code, you'll get direct access to the public properties you define. One of the most complex issues I had to figure out was how to make sure these properties are exposed as the desired datatype. As the CLR code is exposed to both SQL Server and .NET Framework applications, transmogrifying between these types can be problematic. I'll point out the extra code and special attributes I had to add to get this UDT working as it should.

One important point: When you instantiate any SqlClient datatype (SqlTypes), its default value is not 0it's NULL. This can be confusing, as without presetting a value, you'll discover that numbers added to a SqlDecimal don't seem to accumulate. That's because NULL + anything = NULL.

You have some flexibility when defining a CLR UDT, or it would appear so. It turns out that if you choose to store fixed-length component values (like integers or floating-point numbers) that can be stored in data structures that don't expand and contract as values change, the task of creating a CLR UDT is a lot easier. However, if you need to store variable-length values such as strings, SqlDecimal, SqlMoney (and others), or XML in a UDT, you'll have to do the serialization yourself and expose appropriate functions (IBinarySerialize.Write and IBinarySerialize.Read) in the CLR UDT code to do the work. Don't get excited. Serialization can be done quite easily. The GPS UDT implemented next uses fixed-length values and "native" serialization. I'll use "custom" serialization for the typICurrencyV2 UDT built later in this chapter.

Constructing the Custom Property Code

At this point, you'll need to define the properties used to hold the component values. There are several important points here that you should not overlook:

  • I recommend using the Visual Studio Class Diagram tool (shown in Figure 13.55) to help construct the component Properties and Fields. Basically, this tool provides a dialog-based code generator that can make the tedious task of entering all of the component properties easy. One strange side-effect of this tool is that it creates a class diagram and adds it to your projecteach time you click on "View Class Diagram". Before long, you'll have a half-dozen class diagrams in your project. It's best to simply click on the auto-generated class diagram to view its contents.

    Figure 13.55. The class diagram for my initial GPS UDT.

  • Note there are two parts to the CLR Propertythe private "Field" used to store the data and the public Property used to Set and Get the value. When you decide how your CLR UDT is to be defined, you'll need one of each of these for each component. For the UDT, I created eight public properties to hold values for the GPS latitude and longitude values. The UDT template had already created both IsNull and Null public properties.

  • The Property constructed by the Class Diagram tool is not complete. You'll have to visit each property and manually set the Get and Set code. In this case, these are all coded in exactly the same way (as shown in Figure 13.56), so it should not be hard to populate these Property routines.

    Figure 13.56. These properties were partially generated by the Class Diagram tool.

  • The names you assign to the CLR UDT Properties are the names used to reference the component parts of the UDT in T-SQL. No, none of the Visual Studio or SQL Server GUI "RAD" tools exposes these properties, so if you don't have access to the source code, there is no way to determine these property names later.

  • Yes, when you instantiate the UDT in an application, Intellisense does expose the public properties and their associated datatypes.

  • You should try to use SqlTypes in lieu of .NET Framework types, as they don't require transmogrifying as the code navigates between layers. This means better performance and more functionality.

  • Remember that if you want the UDT public properties to support nullability, you'll need to define them using SqlTypes.NET Framework types cannot be set to NULL.

  • The order in which the local (private) memory Field objects are defined sets the order in which the values are stored in SQL Server. This might seem like a "who cares" issue, but when it comes time to debug your UDT, you'll want to be able to know which property is which.

Setting the UDT SqlUserDefinedTypeAttribute

The UDT attributes tell the compiler how the UDT is to be formatted and accessed. These attributes are coded within < > symbols ahead of the Function declaration, as shown in Figure 13.57. Basically, there are three types of "formatting" (set by the Format attribute) when defining UDTs that determine how SQL Server stores and retrieves data in the database column: Native (the default), UserDefined, and Unknown, as detailed shortly. The SqlUserDefinedTypeAttribute attribute is also used to define the StructLayout, which determines how the UDT is serialized (saved and restored from binary). The SqlUserDefinedTypeAttribute properties include:

  • Format: Dictates the serialization format as described earlier. The default is Native, which permits the UDT to store fixed-length value types and implements a very simple serialization formatperhaps too simple for your needs. If your UDT includes reference types (including String), you must specify UserDefined as the Format and implement your own custom serialization routines. Don't be alarmed by this requirement; a serializer/deserializer is not that tough to write, as I illustrate later.

  • IsByteOrdered: Indicates whether the UDT is byte-ordered (in the same way as SQL Server Byte types are ordered in memory). If True, the UDT can create an index on the UDT column. This attribute determines how SQL Server performs binary comparisons on the data. This setting also determines how your custom serializer constructs the binary version of the UDT before you write it using IBinarySerialize.Write.

  • IsFixedLength: Indicates whether all instances of the UDT are defined as the same length (a value = to MaxByteSize. This attribute must be set if you specify a UserDefined format. Again, this value is used by your code to write N bytes as you serialize.

  • MaxByteSize: No UDT can be longer than 8,000 bytes. This is a version-specific requirement that's a function of the size of the SQL Server data page. Having any single column this long is going to be problematic in any number of ways, so if 8,000 bytes is an issue, you need to think about normalization. You don't have to set this when working with Native formatted UDTs.

  • Name: The name used by Visual Studio tools to reference the CLR UDT. I discuss how to code T-SQL to reference your new UDT later in this section. Once deployed, you'll see this name appear in the datatype chooser dialogs in the Visual Studio and SQL Server tools. The name used by SQL Server is defined by the Structure or struct name.

  • ValidationName: This string addresses a method in the UDT to validate the UDT instance when deserialized from an untrusted binary source.

Figure 13.57. CLR UDT attributes and Structure declaration.

[View full size image]

I'll implement the GPS UDT to illustrate how to code most of these attributes.

The CLR UDT also includes the StructLayout attribute that defines how SQL Server should lay out the UDT structure in memory. This structure permits you to set the LayoutKind, CharSet, Pack, and Size. I won't be illustrating these finer points here. Suffice it to say that there are three options:

  • Sequential: In this case (the default), objects are laid out sequentially. You can dictate the packing size using the StructLayoutAttribute Pack value.

  • Auto: In this case, the Framework runtime automatically chooses an "appropriate" memory layout. If you choose Auto, any attempt to expose the UDT outside of managed code throws an exception.

  • Explicit: In this case, you choose how each element of the UDT is laid out in memory using the FiledOffsetAttribute.

As you can see, you have a great deal of control over how the CLR UDT is configured, serialized, and laid out in memory. If you implement a UDT, I suggest you experiment with these options to find the optimal storage formatit can materially impact storage space, indexing performance (or ability), and fetch/store/search performance.

The Public Structure Declaration

The code shown in Figure 13.57 simply names the UDT structure[15] and defines the serialization scheme. It's the name used by T-SQL to refer to the UDT when extracting data or defining a SQL Server column using the UDT. Directly after this structure declaration, you'll likely want to declare your local variables that hold the UDT component elements.

[15] Note that in pre-RTM versions of the 2.0 Framework, the template simply used the Class name and did not define a structure to define the UDT.

The ToString Function

This code is responsible for implementing requests to return the binary UDT data as a string. You decide how the data is to be exposed. For example, if your UDT stores a phone number (country code, area code, prefix, number, extension), you could return a string such as "+011 (425) 555-1212 X123" when the UDT ToString method is called in code. For the GPS UDT example, you store eight parts: latitude and longitude degrees, minutes, seconds, and declination. These are returned as a string in the ToString function shown in Figure 13.58.

Figure 13.58. Constructing a string to display the contents of the GPS UDT.

[View full size image]

The UDT ToString function is called when a T-SQL query executes a CONVERT function to retrieve the column, as shown in Figure 13.64, later in this section.

The IsNull Property

This property is exceptionally simple; it just returns the state of the private null property.

The Null Property

The Null property returns a new instance of the CLR UDT and sets the private null property to True, as shown in Figure 13.59.

Figure 13.59. Managing NULL state with UDT properties.

[View full size image]

The Parse Function

The Parse function is by far the most complex part of the UDT. It's used to parse a formatted string submitted by a T-SQL query and transmogrify the values to the UDT public properties. Typically, this means your code has to convert from a string and save to another datatype. However, in this example, I simply parse a sexagesimal-formatted latitude and longitude string (such as "47° 40' 10" N 122° 3' 26" W") extracting the individual properties. Note that the string shown here in the book uses "smart quotes". This means that the single quote and double quote are different symbols than the ASCII single and double quote symbols. I incorporated this functionality into the UDT parse routine to make it easier to enter a GPS location and not have to worry about doubling up the single quotes. Yes, this is a variation in the "O'Malley" rule that I discussed when working with ADO.NET Command Parameter objects.

The Parse function must walk through the input string (whose format you define) and set the value of each and every component of the UDT. In the GPS example, the Parse routine is particularly long, as it has to pick eight elements out of the string. It's shown in Figures 13.60 and 13.61.

Figure 13.60. Finding the argument markers in the inbound formatted GPS string.

[View full size image]

Figure 13.61. Set the UDT component properties from the string values supplied.

[View full size image]

This code creates a number of integer pointers that are set based on the presence of specific markers for each of the eight component pieces of the latitude and longitude string. For example, you look for the degree symbol after the latitude and longitude degree value. If any of these markers are not found, the routine throws an exception and the operation is cancelled. That's an important part of the Parse function. It must reject any input string that does not comply with the specifications that you define. Sure, you can define as many different input string variations as you choose to support. When I implement the typICurrencyV2 UDT, I'll permit several variations of input string. This permits me to pass strings formatted for older versions to be accepted by the latest version.

When the public properties defined for the GPS UDT are defined as SqlByte, I suggest using SqlTypes throughout to make movement of data into and out of the UDT easier. In several cases, I use the SqlByte.Parse function to convert the string representation of a number to the SqlByte value. If one of the Parse functions fails, the CLR throws an exception and returns control to the SQL Server operation attempting to execute the T-SQL.

Yes, I could also add checks to ensure that the values supplied fall within the range of a proper latitude and longitude location. That is, the degrees latitude must be between 0 and 90, and degrees longitude must be between 0 and 180. While not implemented here for simplicity, I'll show how to add a validation routine to a UDT later in this section.

Debugging the CLR UDT

Because of several serious shortcomings in Visual Studio 2005, you'll find that debugging a CLR UDT under construction is kinda tiresomeand dangerous[16]. I'll try to make it easy for you, but be prepared for an occasional lockup and Visual Studio crashing from time to time. Yes, save your work often. One of the problems you'll encounter is that the IDE always tries to redeploy your UDTeven though it might already be deployed from an earlier test (or an existing version). While SQL Server supports update-in-place for UDTs, Visual Studio does not use or give you the option of using this feature when starting a debug test run. This means you'll need to set up a separate T-SQL script and the means to execute it separate from the Visual Studio IDE. Nope, you can't just set up an "initialize" script and save it as a separate test.sql scriptthat's because the IDE tries to deploy on all scripts.

[16] I lost an entire morning's work when Visual Studio crashed and lost the original project. Save often and back up your work frequently.

To build, deploy, and debug your CLR UDT, follow these steps:

1.
When you're ready to test your UDT, you'll need two test scriptsone that Visual Studio executes to exercise your UDT and another to initialize the server back to a pre-install state. That is, you'll need to completely remove the UDT and any references to it before you can debug your UDT using Visual Studio. I'll show you how to set these up. While there are many alternatives, I like to use an instance of SQL Server Management Studio to execute the initialization/cleanup T-SQL batch. The one I use to debug the GPS UDT is shown in Figure 13.62.

Figure 13.62. The initialization script used to tear down the table and UDT prior to debugging.

[View full size image]

Make sure you designate the test.sql (or your chosen test script) as the default test script using the project menu.

The problem is, Visual Studio always tries to build and deploy just before executing the test.sql test script. While this works the first time (because the CLR UDT does not exist), it fails after that because Visual Studio does not tear down the dependencies on the CLR UDT objectno matter how simple. CLR UDTs can be replaced "in place" like stored procedures and functions, but as I said, Visual Studio does not know how to enable this option. This means that they must be removed and replaced before you try to debug your code. This means you'll have to build a Test.sql script that creates the target database table and inserts rows into that table, as shown in Figure 13.63.

Figure 13.63. The Test.sql test script used to create and populate the UDT table.

[View full size image]

As the complexity of your projects increases, you'll discover that you'll need to tear down every single table, aggregate, function, and stored procedure that references the UDT before you can replace it with Visual Studio. I'm of the opinion that this should be fixedand it might be very soon. Until it is, you might find it easier to simply construct your own deployment scripts that do permit "update in place" (ALTER).

The Test.sql script was created by using the SQL Server Management Studio table designer to define the database table after having created the GPS User-Defined type (typGPS) the first time. I then used the Object Explorer to generate a script for the table and write it to the Clipboard. I subsequently pasted that T-SQL into the Test.sql script executed by Visual Studio in the IDE. Mark this script as the default test script using the project right-click menu.

Note that the INSERT statement in the test script is fairly unremarkableexcept for the long (sexagesimal-formatted) string that defines a GPS location. Yes, the INSERT statement can pass data to the UDT in a number of ways, including passing a hex value. I'll show you how when I implement the typICurrencyV2 UDT later in this chapter.

One of the problems I faced with this approach (of using single and double quotes to indicate minutes and seconds) is that I had to remember to double the single quotes here in T-SQL and double the double quotes in Visual Basic (and C#). Note that I tested with both ASCII and "smart" quotes.

2.
The debug scenario works like this. First, execute the SQL Server Management Studio tear-down T-SQL (as shown in Figure 13.62).

3.
Next, set a breakpoint in your UDT code. I usually break at the top of the Parse function.

4.
Next, in Visual Studio, click the Start Debug button or press F5. This starts the compile, build, and deploy sequence.

5.
Next, Visual Studio starts the Test.sql debug script that's designated as the default debug script. Make sure your script is so designated.

6.
The Test.sql script (shown in Figure 13.63) creates a new table using the just deployed CLR UDT and starts adding rows to it using the INSERT statements.

7.
If the debug process uncovers a bug in your code, you'll have to stop the debugger and start the process again at step 2.

8.
If the UDT works, you're ready to move on to the "select" phase to further test your UDT.

Using SELECT to Query UDT Data

Early in the development of this functionality (during the numerous beta releases), developers had to jump through a lot of hoops to get T-SQL to return data from their UDTs. Along the way, I saw versions that required C++ addressing technique and a number of very strange ways to simply return the data stored in a UDT. The shipping version (thank goodness) makes it easy to get your data back using a SELECT statement.

Figure 13.64 illustrates several ways to use SELECT to return data from a UDT column. The first technique first refers to the "GPS" column and next uses the T-SQL CONVERT function to return the data. The T-SQL CONVERT function calls the ToString function in your UDT so that the data is formatted as your code dictates. If you don't use CONVERT, the data returned is dumped out in hex, as shown in the "GPS" column in the first resultset.

Figure 13.64. Using SELECT to return data from the GPS column.

[View full size image]

This hex notation needs a bit more scrutiny, so I created a detailed view of that data (as shown in Figure 13.65). Note that the SqlByte fields are actually stored as 16-bit values with the leading byte set to "01". If the column element is set to NULL, the leading byte is set to "00". Once you understand how the (all fixed-length datatype) UDT is stored, you can more easily debug and tune your code to optimize its use and database space consumption. As it is, I have been able to store eight (nullable) elements in a data structure that's only 17 bytes long. That's not too bad.

Figure 13.65. Data stored in SQL Server for the GPS CLR UDT.

No, you don't have to use the CONVERT function to return the entire fully formatted GPS location value; you can instead choose to address the public UDT properties or simply supply the raw binary (perhaps to impress one of your coworkers). It could not be much simpler. Each UDT component is addressable by name (the name you chose when you defined the UDT properties) using dot notation, as shown in Figure 13.66. Yes, you might want to CONVERT the raw hex data (as done in the example), but that's up to you and the people (or programs) that have to consume the data. Note that public properties exposed by your UDT can have a NULL value (because I used SqlTypes).

Figure 13.66. Using SELECT to address specific components of the UDT.

[View full size image]

Tuning Your UDT for Performance

Every line of code contained in your CLR UDT plays a role in how well or quickly it performs. First, consider that SQL Server's "native" datatypes also consume CPU resourcesI take that for granted. However, some native types consume more resources than others, and I also take that into account as I design my databases. For example, I chose tinyint over integer or varchar to save disk space and reduce I/O traffic and memory consumption. CLR UDTs are no different. As you add components to your type, keep this in mind. Use the smallest, lightest datatypes possible to get the job done.

There are a number of things you can do to improve UDT performance, but these can mean compromised functionality. The version of the CLR UDT described earlier (let's call it "version 1") permits the inbound GPS location to be formatted in a human-readable format that's easy to visually scan for missing or out-of-place elements. It also supports NULL components, as I used the SqlTypes (SqlByte) to implement the type.

If you choose a simpler (albeit harder to read) input string, you can use the Visual Basic .NET and C# Split function to quickly convert the delimited string to an array of strings that can be written to the UDT components. I implemented this change, and the code is shown in Figure 13.67. It's simpler and smaller (and faster).

Figure 13.67. Using Split in the Parse function improves performance.

[View full size image]

Note that I also changed the UDT to store the CLR public property values in byte objects. The net result of changing to native Framework byte objects means that instead of 34 bytes per column entry, the same data is stored in 18 bytesa hefty savings. However, I did lose the ability to set the public properties to NULL. Yes, you can still set the entire UDT to NULL (assuming you set up the table column to permit NULL values). Since a GPS location does not really make sense if a component is missing, perhaps not supporting NULL components is a good compromise.

Referencing and Updating UDT Values

You can code SELECT statements against the CLR UDT data in a number of ways. However, note that your UDT ToString is called only when using the CONVERT function. If you SELECT a UDT column, as in Figure 13.68 (line 2), without conversion, SQL Server does not call your UDT codeit simply returns the binary representation of the data stored in the database table column. This means that if you expect to reference the UDT public properties or the UDT column value as a whole, you'll need to include code in your project to decode itso any out-of-process application that consumes the UDT binary value must include a reference to your UDT in the project References section. Sure, T-SQL won't have any problem referencing and writing to your database's UDT. However, a Windows Forms, ASP, or Web Service application won't be able to reference it or even see it (in Intellisense) without adding a reference to the UDT structure.

Figure 13.68. Returning data from a CLR UDT using SELECT.

Handling NULL Values

When working with a structure like a UDT that exposes private properties cast as SqlTypes, remember that any one (or all) of these individual properties can be set to NULL. You have to decide if it makes sense that one of the properties has an "unknown" state. Does a GPS coordinate make sense if either the latitude or longitude hours are NULL? Not really. You can get a rough idea of a location given just the hours values without the minutes and seconds, but you still need the declination of both latitude and longitude for the coordinate to make sense. While I didn't go to the extreme of configuring the NULL handling in the GPS UDT in this manner, it's a good exercise for you to try.

If the GPS column value is NULL, depending on your implementation, you won't be able to fetch the data for that row. In this case, you'll have to use the CONVERT routine (Figure 13.68, line 4) that calls your CLR UDT ToString function that interprets NULLs specifically, or, if the type supports it, SQL Server notes the column as NULL. You'll also get an exception "Mutator 'strSymbol' on '<UDT> variable' cannot be called on a null value." if you try to set a public property when the UDT is set to NULL (its default state). Just make sure to initialize the UDT value before trying to set public property values.

Another approach is to set the OnNullCall method attribute. If this is set to False, the UDT returns NULL without evaluating the method when at least one of its input arguments is NULL. If OnNullCall is True (which is the default), the resulting value is determined by evaluating the method, regardless of the argument inputs.

The public properties of the UDT can also be retrieved using dot notation and CONVERT, as shown in the SELECT in Figure 13.68, line 14. When you fetch specific public properties in code, your UDT property Get code is called. Likewise, when you set a value in code, your property Set code is called. Another approach is to invoke the UDT ToString function directly, as shown in line 24. Later in this chapter, I discuss how to reference a UDT from a "normal" (not hosted in SQL Server) application.

Setting UDT Public Properties

While it's possible to use the T-SQL CONVERT function against all of the public properties of a CLR UDT, it's also possible (and tempting) to use the SET statement to set specific public property values. The problem with this approach is that it's expensive, considering that each trip to the UDT code burns CPU resourceseach SET statement makes a round-trip through the SQL Server to CLR UDT interfaces and back. However, there may be places where you feel this works better as when you have an "instantiated" UDT and want to change a single component property.

The other wrinkle here is that there is no "DECLARE X AS NEW Y" in T-SQL. This means you'll face the same object instantiation issues you face when declaring a variable in a CLR language like Visual Basic .NET or C#. You simply won't be able to reference the object without first creating an instance of the object. Because of this limitation, you'll still have to use the CONVERT function to set up the initial CLR UDT "instance" in a T-SQL applicationafter that, you can reuse it and set individual properties using the T-SQL SET statement, as shown in Figure 13.69. Of course, this is not an issue in a CLR language, as you can instantiate an instance of the UDT in code before referencing it.

Figure 13.69. Using the SET operator to set individual CLR UDT component properties.

[View full size image]

The UDT component property names are case-sensitive. This means you'll not only have to know the spelling of the property, but you'll also need to observe the same case as the defined CLR code.

In this case, the code invokes the CLR UDT code four timesonce when I use the CONVERT and three more times when I use the SET statement on the value, symbol, and currency component properties.

Tip

You'll also find that you can't execute an UPDATE statement that sets more than one component property of the UDT.

Stepping Through the UDT Code

When you reference your UDT from T-SQL or some other application, the UDT code might be called many (many) timesor not at all. If SQL Server can figure out what to write to the UDT column without calling your code, it will. For example (as illustrated in Figure 13.70), the UDT code is not called until line 20. At that point (when the code sets the value of the @Price variable), the following sequence takes place:

  1. The Parse function is called with "$50:USD". This creates a new instance of the typICurrency data state and sets its values to the symbol, money value, and currency type provided in the input string. These values are persisted in the private variables.

  2. The Write function is called to return the encoded UDT private variable state to SQL Server. In this case, SQL Server simply saves this in the @Price T-SQL local variable.

Figure 13.70. Setting UDT values.

[View full size image]

At this point, the UDT is stateless. It does not keep any values between invocations. When line 18 is called, the following sequence takes place:

  1. SQL Server calls the UDT Read functionsending a binary stream containing the current state of the UDT value. Read deserializes the binary stream that is generated from the data state of the @Price variable (typed as typICurrencyV2). Read calls the Parse function to re-populate the private variable state. In this implementation, I'm able to take this shortcut because the serialized data structure is the same as the input string that Parse expects. This might not be the best option, so you'll probably want to code a more sophisticated serializer.

  2. The public property (Price.strSymbol, in this case) Set routine is called. This saves the incoming value to the specific private variable.

  3. SQL Server calls the Write function to capture the current state of the private variables.

Remember that a UDT does not maintain stateit can't. The data it knows how to parse and manage is kept in memory or stored in a column value in the database. A UDT might be called by a multitude of different processes, procedures, aggregates, and other server-side executables. Its entire function is to simply decode the values sent to it either in string form (when initially set) or in binary form when referenced from a query that returns a column value.

Yes, it's also possible to set the value of a UDT directly via binaryif you feel brave. This bypasses the ToString initialization step, and it works something like Figure 13.71.

Figure 13.71. Setting a UDT column value with a hex literal.

[View full size image]

Validating Your UDT Data

The process of storing your UDT data into the database is (of course) managed by SQL Server. It takes certain precautions to ensure that the data is valid and meets the serialization criteria, but if there are business rules or other criteria that you wish to impose on the data, it's up to you to implement those tests. The Framework exposes a mechanism for this, and it's hooked up when you first define your UDT. Unfortunately, it's not called very oftenapparently, it's called only when SQL Server has to do a "conversion," as when you convert from bytes or when a UDT comes in from an external source. The validation routine is not called when you call the CONVERT function or execute INSERT or UPDATE. This means you'll have to add it to your own Parse routine to do a last-second validation test.

The ValidateGPSData function tests each component of the current UDT instance and returns False if it does not comply with the rules dictated by accepted latitude and longitude conventions. This routine and every other routine in your UDT code works on an instance of the UDT passed to it from SQL Server as it processes T-SQL statements. This means that several instances of this code could be running at onceeach processing its own instance of an in-progress UDT. Visual Studio and the language compilers make sure that the code conforms to this instance-centric approach.

To "activate" the validation routine, you'll need to set the ValidationMethodName in the UDT attributes, as shown in Figure 13.73.

Figure 13.72. Validating your UDT data structure using a custom function.

Figure 13.73. Activating UDT validation.

I put a call to the ValidateGPSData function just before returning the populated UDT instance in the Parse function, as shown in Figure 13.74. Your test script should include tests for each leg of your validation routine, to ensure that they all work.

Figure 13.74. Testing the UDT structure just before returning it to SQL Server.

[View full size image]

Implementing UDT Methods

You might feel the need to implement methods on your new UDTespecially since the Visual Studio UDT template creates a couple examples for you. The question is, do you want to implement additional code that might better be implemented elsewhere? Remember, each change to the UDT requires a complete tear-down of the column (including any data it contains), redeployment of the tested UDT binaries, and repopulation of the table.

I implemented a simple method (as shown in Figure 13.75) to illustrate how to code and invoke UDT methods. Our method simply returns a string that describes the string format used to describe a GPS location.

Figure 13.75. Implementing a simple CLR UDT method.

[View full size image]

Calling the method is also simple (sort of). As shown in Figure 13.76, without the TOP clause, it returns a row for each row in the GPSLocationV2 tablewhich is understandable since I didn't include a WHERE clause in the query.

Figure 13.76. Calling the UDT QueryTemplate method.

Setting Custom Method Attributes

Hidden in the documentation, you might stumble across references to additional method attributes that can be required in some cases and simply helpful in others. These attributes are exposed by the SqlMethodAttribute property, and I've already discussed a few of them. For example, when you create a Table-Value Function (TVF), you must set the FillRowMethodName to identify the method used to return rows. The other property settings are shown in Table 13.2.

Table 13.2. SqlMethodAttribute Property Settings

Name

Default

Description

DataAccess

If set, the function accesses user data in local SQL Server instance. This attribute is required for any Function or Stored Procedure that needs to open a connection. This attribute is not available on CLR UDTs or Aggregates.

FillRowMethodName

The name of a method in the same class as the table-valued function (TVF) that is used to return rows used to populate the table.

InvokeIfReceiverIsNull

False

If set, SQL Server invokes the method on a null reference instances. This is another approach to handling special NULL case situations.

IsDeterministic

False

Indicates whether the user-defined function is deterministic. "Deterministic" means that the routine always produces the same result, given the same inputs. For example, the function Len returns a different value each time it's called, but the same value when a 10-byte string is testedit's deterministic.

IsMutator

False

Indicates whether a method on a UDT is a "mutator"a method that changes a component property in the UDT.

IsPrecise

False

Indicates whether the function involves imprecise computations, such as floating point operations. This property determines if computed columns that reference this function can be indexed.

Name

The name under which the function should be registered in SQL Server. Used by the Visual Studio IDE.

OnNullCall

Indicates whether the method on a UDT is called when null reference input arguments are specified in the method invocation.

SystemDataAccess

False

Indicates whether the function requires access to data stored in the system catalogs or virtual system tables of SQL Server.

TableDefinition

A string that represents the table definition of the results, if the method is used as a table-valued function (TVF). It's used by Visual Studio only to automatically register the specified method as a TVF.

TypeId

When implemented in a derived class, gets a unique identifier for this Attribute.

In the next section, I'll introduce SqlFacet, which is a method or property attribute that's used to configure the datatype being used to return data from a function, method, or attribute. I use SqlFacet to ensure that the SqlBinary Precision and Scale properties returned by the UDT value are coerced to the correct settings.


Implementing Advanced CLR User Defined Types

I wanted to get you grounded in some fundamental principles before wading into some of the more advanced concepts involving UDTs. Yes, UDTs can be very code-intensive and can introduce a number of more complex issues. As I worked with the typICurrency UDT in this section, I discovered a number of nuances that are not covered in the documentation but are critical to the ability of the UDT to correctly manage the data as intended.

Woven into the fabric of this section are the following topics:

  • Coding your own CLR UDT serialization routines: These can permit you to implement versioning on the UDT. This means when your pointy-haired boss decides that it's time to upgrade or tune your UDT, you won't need to throw out existing datayou simply bump the UDT version. To implement this feature, the custom serializer I'm about to show you knows how to morph the data so the old data is automatically converted into the new format. This is a feature that existing SQL Server (aliased) User Defined Types could not support.

  • Building your own deployment scripts: Because of the limitations in Visual Studio 2005, I found it easier to build a custom T-SQL script that performs the deploymentthis way, I don't (always) have to tear down all dependencies as each new version is built.

  • Accessing your new UDT from other applications: This section discusses how to set up a Windows Forms application (or any other type of application) that incorporates the UDT into the registered classes visible to the compiler. I'll show you how to pass a UDT from tier to tier and in arguments passed to functions, methods, subroutines, or local properties.

  • Using a UDT in strongly typed data structures: Since the Visual Studio IDE does not directly support UDTs, you'll have to take a few extra steps to permit you to create strongly typed data structures that leverage this technology. I discuss a couple of easy tricks to accomplish this.

For the example in this section, I'm going to start over with another UDT designed to manage international currency values. It's a variation of the code Bill used to demonstrate CLR UDTs in the years leading up to SQL Server 2005's RTM: typICurrency. The folks at Microsoft seem to think it's a good example, toothey implemented a lightweight version of it for their example in the MSDN help topics, but I'm going to take that implementation up a few notches.

The typICurrency UDT is designed to manage currency (money) values, where values can be in any currency denomination. For example, typICurrency can store dollars, euros, British pounds, or any currency type in the same column. The UDT keeps track of the currency denomination and other helpful values, like the currency symbol, to make selecting and displaying the data easy. I initially implement this UDT using strings and variable-length datatypes, so I'll have to use custom serialization.

User-Defined Serialization

Before long, you're going to outgrow default (Format.Native) serialization. Keep in mind that if your UDT stores values can be persisted as bool, byte, sbyte, short, ushort, int, uint, long, ulong, float, double, SqlByte, SqlInt16, SqlInt32, SqlInt64, SqlDateTime, SqlSingle, SqlDouble, SqlMoney, or SqlBoolean, it's best to simply stick with the default serialization. That is, unless you want to implement custom versioning, which permits UDT data to be identified by version. This way, you can upgrade from one implementation to anotherautomatically morphing the data as you go. I'll show you how to implement custom versioning later in this section.

If you use default serialization, you should not (cannot) specify MaxByteSize, and all of the component parts (the fields) of your UDT must be serializable (all of the datatypes mentioned above are serializable). SQL Server also forces you to lay out the items in your UDT structure sequentiallyI saw how to set this order earlier in this chapter. This regimen can seem a bit stifling, and there is a viable alternative: user-defined serialization.

Okay, so you want to expose an unsupported type like SqlBinary, SqlString, or some other variable-length public property value in the UDT. This complicates things a bit, as now you have to turn off the autopilot and move your data into and out of SQL Server without benefit of the automatic formatting and serialization. At first, this might feel like your first solo flight in an OH-23 helicopter with 8 hours of trainingbut not nearly as life-threatening.

To start with, you'll need to specifically set the following UDT attributes when you first declare your UDT Structure (or struct):

  • Format: This must be set to "UserDefined" when you intend to code your own serializer. This tells the compiler (and SQL Server) to look for the IBinarySerialize interface in the UDT. Once set, SQL Server calls the Read method to retrieve the binary data and the Write method to save it back to the data table column.

  • Name: This attribute simply names the UDT so it can be managed with Visual Studio.

  • IsByteOrdered: This tells SQL Server how you have stored the data and helps SQL Server manage binary comparisons on the UDT[17]. In other words, IsByteOrdered=True says that the bytes are written in sequencejust as they are when working with managed code. If you want the UDT to support comparison operators, GROUP BY, ORDER BY, and PARTITION BY, you have to set IsByteOrdered to True. In this case, SQL Server uses the disk representation of the instance in binary comparisons. Operators supported include =, !-, <, <=, >, and >. The default is True.

    [17] For more information, see http://msdn2.microsoft.com/en-us/library/microsoft.sqlserver.server.sqluserdefinedtypeattribute.isbyteordered.aspx

  • IsFixedLength: You'll probably want to set this attribute to False when using variable-length data types. However, a fixed-length structure that contains variable-lengths components is possiblebut tough to implement.

  • MaxByteSize: When using the UserDefined UDT format, you'll have to set the MaxByteSize attribute to the combined size of the UDT components in its serialized form. That is, MaxByteSize tells SQL Server the size of the longest data stream your Write routine returns. Since you're in charge of implementing the serializer, this is a value you determinebut it might be a bit tough if your UDT has a number of variable-length components. For example, if you have a single System.Char column that's 10 bytes long, it consumes 22 bytes, 16 bits/character (to implement the Unicode UTF-16 character), and 2 bytes of overhead used while serializing. Just remember to include the entire data length when computing MaxByteSizethe data plus the serialization overhead. The upper limit is about 8,000, but you don't really want to save that much data in any single column; considering the maximum size of a data row is about 8,000, having a couple of columns consume all of the space flies in the face of most relational theory.

Building the iTypCurrencyV2 UDT

Once your structure attributes are set (which is probably the last thing you'll set after the UDT is written), you need to add the IBinarySerialize Implements annotation, as shown on line 12 of Figure 13.77. This is used to expose the standard interfaces where the Read and Write methods are implemented. These are used to accept and return data from and to SQL Server.

Figure 13.77. The attributes and structure declaration for typICurrencyV2.

[View full size image]

Note that I'm implementing "version 2" of the UDTtypICurrencyV2[18]. Version 1 is a good place for you to start exploring how to code a UDT. Version 2 goes quite a bit farther in the implementation to enable many of the more complex features you might need in a UDT.

[18] A basic version (typICurrency), as well as this advanced version, is included on the DVD.

The UDT attribute and structure declarations for the new UDT are shown in Figure 13.77.

When you specify which interfaces to implement, be sure to let Visual Studio help you with Intellisense. If you do, Visual Studio automatically adds the implementation prototype routines in your code.

The code used in the typICurrencyV2 UDT is different enough that I'm going to step through it section by section to let you see the approach I use at each phase. One aspect of the user-defined (hand-rolled) serialization is that if you aren't careful, the UDT data written to the server tends to consume more memory, more I/O channel capacity, and more disk space. This overhead can consume more resourcesperhaps more resources than the implementation is worth in increased functionality. I coded the UDT to keep the size of the stored data and the amount of code needed to manage it to a minimum. No, writing a serializer is not hard. It might seem daunting at first, as you have to decide what's being written (in binary) to the table column. I'll show you how to inspect what you've written to the column value so you can tune it for size and functionality.

Declaring and Parsing the Private Variables

The private variables declared in the UDT define how the UDT data is stored (temporarily) within the UDT. No, they don't have any bearing on how the data is stored in the database or how it's exposed to consumers (those applications referencing the UDT data). The private variables and the supporting public properties that Set and Get them were templated with the Visual Studio Class Diagram tool. In this case, I chose SqlString variables to manage the currency symbol ("$") and the currency type ("USD") values. I could have used a string for the money value as well, but I wanted to illustrate how to handle a more complex type within the UDT. This implementation (which is different from the "version 1" basic typICurrency implementation) supports a Version property that's used to help morph data from one implementation footprint of the UDT to the next. Version 2 also knows how to recognize version 1 formatting so it can automatically upgrade to version 2 notation.

Tip

Remember, I discussed each of these UDT support routines earlier in this chapter.

Coding the iTypCurrencyV2 ToString Function

Note that the UDT ToString function implementation uses String.Format to return the string representation of the current UDT private variable stateassuming the UDT IsNull property persisted as m_Null is set to False. Again, this output format does not have to match the input format used when you set the UDT value with the CONVERT statement, but it helps. As a matter of fact, the string you return is not stored anywherejust used to display the contents of the UDT when a SELECT is executed or your code invokes the typICurrencyV2.ToString method directly. Note that the ToString function does not return the Version property.

Figure 13.78. Declaring the private variables and the ToString function.

[View full size image]

The UDT Parse Function

The Parse function's job is no different here than when you use default serialization. The primary goal is to convert the string passed from SQL Server and set the private property values. Yes, the Parse should make sure that these values meet minimum validity requirements, and, no, this routine does not do thatit blindly accepts any currency name, symbol, or type and any value. However, for typICurrencyV2, I added a few extra lines of code to deal with an unknown (NULL) currency type and alternate versions.

In this implementation, I depend on the CLR Split function to parse the string passed to the Parse function (see Figure 13.79, line 61). This deals with several string-parsing issuesin this case, all I need to do is separate the individual UDT values in the inbound string with a colon (":").

Figure 13.79. The typICurrency Parse function.

[View full size image]

Coding the IBinarySerialize Read and Write Subroutines

Because the UDT code is now responsible for serializing the instance data, you need to construct subroutines to convert the private variable data to a data stream (Write) and reconstitute the private variables from that stream (Read). A few random thoughts come to mind here as you define your Read and Write routines:

  • Just exactly how you define the saved data stream is up to you, but consider that every byte you write to the stream is saved to the database table column containing the UDT. No, what I implemented in typICurrencyV2 is not particularly efficient, but I didn't want to confuse the issue by implementing a complex serialization scheme.

  • It's up to the Read routine to decode the stream extruded by the Write routine, so if you make the stream complex, the decoding algorithm must be equally complex to unscramble it.

  • If you serialize complex variables like SqlBinary or SqlMoney, the complexity of the Write and Read routines increases dramatically and you might also end up saving more data than you need. Consider that a SqlMoney datatype can consume from 5 to 17 bytes of space, while the value you store might be as small as 3 bytes ("0.0") when persisted as a string.

  • Each stream that's sent to your UDT for decoding is prefixed with a "header" byte that indicates the NULL state of the UDT binary value. The 1 bit is set if the value is NULL. In this case, Write should simply return Null (as illustrated in the code).

  • In the Write routine, I save the length of the string to be serialized followed by the stream data. This makes it easy to capture the stream in the Read routine where I simply read the length and pass that value to a fixed-length Read function. I added a bit of code to write the length as an (8-bit) byte instead of an integer, as I know the length of the data I'm saving won't exceed 255 characters. This approach saves 24 bits of space per entry.

The UDT Write routine is shown in Figure 13.80.

Figure 13.80. Implementing the IBinarySerialize Write routine.

[View full size image]

Yes, it's probably best that you design and code the Write routine before starting work on the Read routine. The Read routine (as shown in Figure 13.81) is passed an open BinaryReader accessible via Read methods. Again, I have to test the header to see if the stream is NULL, in which case I simply do nothing.

Figure 13.81. The UDT Read routine.

[View full size image]

Our task in the Read routine is to populate an instance of the typICurrency and fill in the UDT instance private variables. I started by reading the byte containing the length of the persisted stream and follow that by reading the entire stream into a string. Another approach might have been to use the type-specific Read methods to directly populate the individual private variables, but this added unwanted complexity. One benefit of this approach is that I didn't have to repeat the code used to parse the inbound stream (I used the same input string format); I simply instantiated a new instance of the typICurrency class and called the Parse routine to flesh it out.

Basically, that's it. As you can see, user-defined serialization is not that complex. I'm now ready to test the new UDT.

Note that all of the routines have their own Try/Catch exception handlers. I found these essential when trying to figure out what part of the UDT code had problems.

Testing the typICurrencyV2 UDT

The testing strategy here is no different than for the earlier UDT examples, but the scripts are a bit different[19].

[19] All of these projects and test scripts are provided on the DVD.

Ah, can I mention a few more points before moving on? Note that that Visual Studio 2005 Data Source Configuration Wizard does not support tables that contain UDTs. No, this is not the end of the world, but it does mean that you won't be able to create strongly typed DataTable or DataSet objects against tables containing UDTsyou'll have to use untyped data structures. And no, you can't use the DataSet designer to build an STD[20], either. SorryI guess it was a bit much to ask. Of course, you can create SQL Views that expose the UDT properties as columns. In this case, much of the code-generator functionality in Visual Studio can be used, but only to create read-only STD structures.

[20] STD: In this case; it's not a sexually transmitted disease; it's a Strongly Typed Data structure.

One other tiresome issue is that the Visual Studio IDE and SQL Server Management Studio do not know how to expose the public properties of a CLR UDT. This means that unless those developers being asked to consume the UDT have access to the source code (as if), they won't have any way (outside of using Reflector[21]) to see the private properties you define to access the specific component properties of the UDT. I guess you had best document it and keep the documentation in sync with the deployed version.

[21] www.aisto.com/roeder/dotnet/

Figure 13.82. The Visual Studio Data Source Configuration Wizard chokes on UDTs.

Previous Page
Next Page


Accessing CLR UDTs Elsewhere

Okay, most of us write applicationsnot just T-SQL stored procedures and the like. This means I'm going to need to access CLR UDTs in a number of ways from my Windows Forms, ASP.NET, and Web Service applications. Before I wander too far away from the UDT discussion, I really need to discuss how you can leverage your shiny new UDT from "elsewhere"outside of a T-SQL code block.

To illustrate the issues, I created the LoanRequest sample application that's used later on to fire a UDT trigger. It accesses the Products table via a parameter-driven stored procedure (naturally) and permits the user to select one of the "products" by lot and attempt to borrow money to buy it. LoanRequest also executes another stored procedure that accepts a typICurrencyV2 input parameter. I'll show how to "prepare" the Parameter object so that that it's recognized by ADO.NET. Along the way, I show how to reference variables and their public properties instantiated as a UDT.

Referencing a UDT

The following steps walk you through the process of creating this sample application:

1.
To get started, create a Windows Forms application using Visual Studio. Sure, this same approach works with any type of application.

2.
Next, click on the "Show All Files" icon in the Project explorer. This exposes the "References" list.

3.
Right-click on References and click "Add Reference". This exposes the "Add Reference" dialog in Visual Studio (as shown in Figure 13.83). If the typICurrencyV2 DLL is not shown in the "Recent" or "Projects" tab, you'll have to dig for it. Click on the "Browse" tab and drill into the bin directory of the project used to construct your UDT. You're looking for the UDT DLL built by the Visual Studio build process. Once you find it, click OK.

Figure 13.83. Browsing for the typICurrencyV2 DLL.

By adding this reference, you're simply pointing to a DLL that's been deployed on SQL Server and visible through access to the database where it's in use.

4.
You might want to add an Imports or include (C#) to your project to assist Intellisense in finding the referenced UDT.

Instantiating a UDT Variable

When you declare a program variable, the typICurrencyV2 appears in the list of visible datatypesassuming you've added a reference to the UDT DLL to your project. Yes, you'll have to prefix the reference with the "project" (assembly) name, as shown in Figure 13.84.

Figure 13.84. Declaring a variable as the CLR UDT typICurrencyV2.

[View full size image]

If you forget to prefix the assembly name, the auto-correct feature of Visual Basic .NET can fill it in for you (as shown in Figure 13.85).

Figure 13.85. Auto-correct can fix the reference to the UDT.

[View full size image]

As you write code to reference the public properties exposed by your UDT, Intellisense kicks in again to let you pick from a list of visible UDT properties and methods (as shown in Figure 13.86).

Figure 13.86. Intellisense exposes the UDT public properties.

IMHO

This Intellisense dialog is the only way to determine the public properties of a CLR UDT, short of using Reflection.

Once you've instantiated the UDT-typed variable, you can fill it from an untyped Object returned from a DataTable by using CType (cast in C#) to coerce the datatype, as shown in Figure 13.87. Once the UDT-typed variable is populated with data, you can pick off the public properties by using normal dot notation.

Figure 13.87. Instantiating a CLR UDT-typed variable and accessing its public properties.

Walking Through the LoanRequest Code

To set up the SqlCommand instance used to execute the AddLoan stored procedure, I need to configure the Parameters collection to accept my UDT value. Since AddLoan expects one of the input parameters to be cast as typICurrencyV2, I'll have to set the Parameter datatype to the UDT type. This is done in two steps, as shown in Figure 13.88. On line 41, I set the SqlParameter datatype to SqlDbType.Udt. On line 42 I set the SqlParameter.UdtTypeName to my UDT type's name. Note that I use the fully-qualified name "dbo.typICurrencyV2", as I did when I declared T-SQL variables. Now I'm ready to set the SqlParameter value in code without having to worry about misunderstood references or type coercion issues.

Figure 13.88. Configuring a SqlParameter to accept a CLR UDT.

[View full size image]

It turns out that the AddLoan stored procedure asks for two different values typed as typICurrencyV2. Both the @LoanAmount and the @PaidToDate SqlParameter instances require special UDT handlingor do they? For the @PaidToDate SqlParameter, I simply pass a varchar large enough to hold the string-representation of the UDT value.

Figure 13.89. Configuring a UDT typed parameter to accept a string.

[View full size image]

To set up the parameter values, I need to do a bit of object instantiation work, as shown in Figure 13.90. In this code, I instantiate a new variable (typICPaidtoDate) as typICurrencyV2, as I want to use the same currency configuration but 0 as the money value. Note how the public properties can be addressed using dot notation (lines 138 and 139).

Figure 13.90. Preparing values to be applied to the stored procedure parameters.

[View full size image]

Once the variables containing the typed values are configured, I can set the stored procedure input parameters, as shown in Figure 13.91. Note that on line 154, I simply use the UDT's own ToString function to return the string format of the typICurrencyV2 value and use it to set the input SqlParameter value.

Figure 13.91. Setting the input parameter values and executing the stored procedure.

[View full size image]

Now that I've ventured off the road to better understand the subtleties of UDT management and coding, I'm ready to take that UDT to an even higher hill and throw it off to see how well it works in an accumulator.


Implementing CLR Aggregate Executables

The next challenge you might choose to scale is building CLR aggregates. These are a requirement if you expect to get meaningful aggregated information from a column declared with a UDTespecially with a UDT such as the international currency typICurrency I've been discussing. Sure, you can still use the built-in T-SQL aggregators (like SUM, MIN, MAX, etc.), but when used against a column whose value is not really stored in the same terms as other values in the same column, the results won't be of much use. For example (as shown in Figure 13.92), you can execute a T-SQL SUM against the Products table in my UDTCurrency database. However, in this case, the sum aggregates U.S. and Australian dollars, yen, British pounds, and euro values without regard to their currency. Most banks outside of Texas don't use this kind of math.

Figure 13.92. Using a T-SQL aggregate against a CLR UDT column.

To solve this problem and any problem that requires a custom aggregate function (whether or not it's used against a UDT, you'll have to write your own custom CLR aggregate function. Again, this can seem like a very big challenge at first, as there seem to be a lot of details to worry about. Relax; I've blazed a pretty clear trail (which mostly involved figuring out how to actually implement the CLR aggregate, given the state of the Microsoft documentation and examples).

Unlike a UDT, coding, building, deploying, and debugging a CLR aggregate is really pretty easythat's because Visual Studio can correctly tear down and use T-SQL ALTER to replace an existing CLR aggregate. This means you'll be able to set up a test.sql script in Visual Studio and simply click F5 when you need to build, deploy, and walk though the codeeach time you need to test it.

What Is an Aggregate?

An aggregate is a block of code designed to "aggregate", accumulate, or total values presented to it. It can also be used to perform set[22] functions on the values supplied. For example, an aggregator can find the maximum value of a set of numbers or the minimum length of a set of strings.

[22] Set as in set arithmetic or set logic.

An accumulator can also format values and return a concatenated result in any format you choose. A CLR aggregate can perform any operation you choose on a given set of values. This means your "SUM" can include or exclude any values you choose. Yes, this smacks of the accounting done at Enron, but I doubt they used a CLR to do the work. Perhaps, they would not have been caught if they had.

When writing your CLR accumulator, you define the input and output data typewhich can be different. In the case of the accumulator, I also define (and coerce) the Precision and Scale to ensure that the data is correctly formatted when sent back to SQL Server. SQL Server (and T-SQL) does the work of picking off the values to aggregate and calls the Accumulate function to pass values to be aggregated, the Merge function to combine previously constructed "sub-totals", and the Terminate function to return the final result. It's the Terminate function that determines the datatype of the value ultimately returned by the aggregate. I'll focus quite a bit on this function.

Walking Through the CLR Aggregate Example

The example aggregate illustrated here is designed to total the mnyValue property values from a column defined with the CLR UDT typICurrencyV2. Before adding any non-U.S. currency to the total, the aggregator code converts the value to U.S. dollars. No, I'm not suggesting that the approach used here is a best practice. For one thing, it depends on fixed conversion rates, which would be suitable for ratios or conversion factors that don't change at all, but for international currency, exchange rates vary quite a bit over time. Perhaps a better example might be a UDT that aggregates temperature values in Fahrenheit, Celsius, and Kelvin and returns the values in a common scale.

I also tried to read the conversion rates from a rates table stored in the database. The problem with this approach is that a CLR aggregate cannot access the local context connection. Yes, it would be possible to open a "normal" shared memory provider ADO.NET connection, but that would significantly impact performance. However, it might be a viable option for some situations where you really can't make do with a CLR Function.

The MSDN documentation (available online or via Visual Studio help) is pretty good for aggregates. It is missing a few (too many, I think) key points that are emphasized here to keep you on track. However, I do recommend that you use the help topics for more in-depth information on attributes.

A CLR aggregate has a unique configuration of functions and attributes that makes it quite different in a few respects from a CLR function or stored procedure. These CLR aggregate entry points (as described below) are called by SQL Server to initialize, add selected values to your in-memory accumulator, and retrieve the final sum.

Let's take a high-level look at the code before venturing into the aggregator's back alleys and underground passages. Figure 13.93 shows the completed CLR SUMUSD aggregate program collapsed to the routine level.

Figure 13.93. The Visual Studio CLR Accumulator application.

[View full size image]

Note that there are several routines each CLR aggregate must support and a few others that you might need to add:

  • The Function attributes: These attributes, shown in Figure 13.93, define how the aggregate is seen by Visual Studio and built on SQL Serverat least, to an extent[23]. Here is where you indicate whether or not you want SQL Server to serialize the data values passed to and from the aggregate, the maximum size, and several other attributes settings that are well documented in the MSDN documentation. Because I'm working with the SqlDecimal datatype, I must provide my own serialization. As you'll see, this is not at all hard, as I'm serializing only one value. The function prototype does not set any of these attributes and defaults to Native serializationthat might be just fine for a simple aggregate.

    [23] The Return type is not set until you get to the Terminate function (as I'll show you later).

  • The Init Sub: This subroutine (prototyped by the Visual Studio CLR aggregate template) is coded to initialize the accumulator state. It's called once when the aggregate is first invoked. For this example, the internal state includes the local accumulatorthe value used to keep a running total of values declared as SqlBinary. Note that I set the Precision and Scale of the private variable sDecAccumulator (line 27). No, this might not be necessary in all cases, but I wanted some control over the way the decimal values were being kept in memory. I chose to use SqlDecimal to give me more control over the specific Precision and Scale value returned from the accumulator.

    Next, I populated the list of conversion rates. I already discussed how it might be necessary to populate this table dynamically. The completed Init routine is shown in Figure 13.94. The AddRate function is home-grownit simply populates a collection of currency types and rates.

    Figure 13.94. The CLR aggregate Init function.

    [View full size image]

  • The Accumulate Subroutine: This routine, shown in Figure 13.95 (prototyped by the Visual Studio CLR aggregate template), is used to accept a typed value and "add" it to the set of values being accumulated. This might mean concatenating the value to a string, adding it to a collection of values to be summarized later, or anything else you can imagine. This routine is called each time SQL Server finds another value to include in the accumulator value set.

    Figure 13.95. The Accumulate function used to gather values.

    [View full size image]

    For this example, the CLR aggregate is designed to accept a (non-NULL) value typed with the CLR UDT typICurrencyV2. The routine tests the currency "type" (USD, AUD, GBP, etc.), and if it's not U.S. dollars ("USD"), it converts the value to U.S. dollars and adds it to the local accumulator (sDecAccumulator) used to keep a running totalU.S. dollar values are simply added to the total without conversion. Note that I make sure the converted value remains cast to the correct Precision and Scale (15,6) by using the ConvertToPrecScale SqlType method.

    The hand-made ConvertToDollars support function, shown in Figure 13.96, uses the conversion rate tables to figure the exchange rate value. Note that I also illustrate a couple of techniques to force the correct Precision and Scale. On line 92, I add a SqlFacet attribute to the Function. This gives the developer a way to specifically define how the return value is to be formatted. In this case, I set the SqlDecimal Precision and Scale. On line 97, I force the result to conform to this same (limited) precision and scale. Without these restrictions, the default behavior explodes the scale with more "accuracy" than makes sense (even to a mathematician).

    Figure 13.96. The custom ConvertToDollars function.

    [View full size image]

  • The Merge subroutine: When SQL Server has to build totals and sub-totals from accumulated values, it calls the CLR accumulator's Merge routine to add these intermediate values to the set of values being accumulated. The routine (prototyped by the Visual Studio CLR aggregate template) is very simple in this caseI simply add the value to the local accumulator.

    Figure 13.97. The Merge routine is used to combine intermediate results.

    [View full size image]

  • The Terminate function: The Terminate function (prototyped by the Visual Studio CLR aggregate template) is designed to summarize the set of values passed one by one to the accumulator and return that value back to SQL Server. This might mean computing the minimum, maximum, average, or other statistical summary value from the set of values. In this example, I simply pass back the current value of the accumulator used to hold the sum of the values presented.

    An important point to note here: By default, if you build a CLR aggregate, Visual Studio deploys the CLR aggregate using default numeric or string values. For example, if you simply declare that the UDT is to return a SqlDecimal without forcing the Precision and Scale, Visual Studio generates a RETURNS[numeric](18,0) clause in the CREATE AGGREGATE statement. This means that if you intend to pass back a floating point number or any number with a fractional component, all you'll get is the whole numberthe fraction is stripped. Strings are also formatted as NVARCHAR(4000), so you might also want to hard-cast the length of these return values.

    To make sure the entire value is sent to SQL Server, I force the Precision and Scale of the return value passed back from this routine using the obscure (but very handy) SqlFacet attribute, as shown in Figure 13.98. This "As" syntax that references the SqlFacet attribute is not documented in MSDN helpit was shown to me when I had trouble finding out how to force the compiler to set these datatype attributes.

    Figure 13.98. The Terminate function illustrating use of SqlFacet.

    If you want to verify what type and precision is being used to handle your CLR aggregate, script the deployed aggregate using SQL Server Management Studio, as shown in Figure 13.99.

    Figure 13.99. Scripting the deployed aggregate in SQL Server Management Studio.

    [View full size image]

  • The Serializer and Deserializer: Because I'm not using one of the simpler types, SQL Server requires that I add my own serialization code. Since I don't have to worry about formatting or reformatting the values, my Read and Write routines are brutally simple, as shown in Figure 13.100.

    Figure 13.100. The Read and Write routines serialize and deserialize the CLR aggregate value.

Testing the CLR Aggregate[24]

[24] The complete example is available on the DVD. See prjAggregateSumUSD.

The bulk of the work required to set up a test scenario has already been doneI did it when I built the test scripts for the CLR UDT typICurrencyV2. When I'm ready to test, I set up a simple test script. Yes, I was able to use the test.sql script sample generated by Visual Studio as a starting point. Our test script (shown in Figure 13.101) expanded on that quite a bit. I suggest using small amounts of test data that include values at each end of the possible values. Once you're happy with the results of the limited test (and the computations do reflect reality), you should move on to larger sets of data.

Figure 13.101. The test.sql script used to exercise the CLR aggregate code.

Because Visual Studio and SQL Server can work together to alter the CLR aggregate in place, it's easy to debug the routine just as you would debug a CLR function or subroutine.


Implementing CLR Triggers

Many, if not most, of the SQL Server databases in production today include one or more triggers. These form the last bastion of defense for your data. A trigger is simply a stored procedure that's executed automaticallyyou decide when. Since the earliest versions of SQL Server, triggers have been programmable to fire when new data is inserted, updated, or deleted on a specific table. In SQL Server 2005, the role of triggers has been greatly expanded to include data manipulation language (DML) operations of all kinds. It's really beyond the scope of this section to get into these advanced types of triggers, so I'll just stick with FOR INSERT, UPDATE, and DELETE triggersbut written in a CLR language.

For this example, I added a FOR INSERT, UPDATE trigger on the Loans table updated by the sample LoanRequest application, discussed earlier in this chapter. The stored procedure called in that example executed an INSERT to the Loans table. One of the business rules I imposed was that no loan could be accepted (or altered) that has a payment below a certain amount. If the payments are not high enough, the loan is rejected and the transaction rolls back. If the loan is accepted, both the Loans and Products tables are updated to reflect the sale of the property lot. In order to determine whether the loan payment is high enough, I had to incorporate a block of code that executed one of the .NET Framework's financial functions (pmt). This function, given a set of loan constraints, figures out how much the monthly payment would be. No, this functionality is not available in T-SQL, so it had to be implemented in CLR code or somehow cobbled together in T-SQL at considerable cost.

Because the trigger needs access to the current transaction state, I must add a reference to System.Transactions.DLL to the project. I also added an Include to help reference these classes at design time.

Building the CLR Trigger

You know the drill. Building a new trigger is really no different than building any of the other CLR executablesat least, it starts out the same. Of course, when creating any new SQL Server CLR executable, you'll have to specify the database where this UDT is to be deployed. Choosing "Add New Item | Trigger" creates a new template that includes a number of attributes specific to the trigger (as shown in Figure 13.102).

Figure 13.102. The unaltered CLR Trigger template.

[View full size image]

As with any other application, you'll have to add a reference to any CLR UDTs to the project so that they can be accessed by the compiler. In this case, I added a reference to the UDT typICurrencyV2.

These attributes are used to give the compiler and SQL Server enough information to build and deploy your UDT trigger:

  • Name: This names the trigger. That's not particularly important, as this name is referenced only from within SQL Server.

  • Target: This attribute points to the database table where the trigger should be deployed.

  • Event: This attribute indicates which events fire the trigger. I'll set this to FOR INSERT, UPDATE so that the trigger fires as new rows are added or changed. For this example, I don't care about the other operations.

The code implemented to be fired when the target table (Loans) is changed by an INSERT or UPDATE operation is not that complex. I start by picking up the SqlTriggerContext, which tells me which of the bazillion operations have occurred. Theoretically, the trigger code should not be called unless the TriggerAction property is set to TriggerAction.Insert or TriggerAction.Update. I use a Case statement (as shown in Figure 13.103) to route execution to the appropriate routine based on the TriggerAction.

Figure 13.103. Routing execution in the trigger based on the TriggerContext.TriggerAction.

[View full size image]

Accessing the INSERTED and DELETED Pseudo Tables

One unique attribute of any trigger that monitors the FOR INSERT, DELETE, or UPDATE operations is the INSERTED and DELETED pseudo tables. When SQL Server executes an INSERT statement, it saves the contents of the new row to a pseudo (in-memory) table that can be tested in your trigger code. Many DBA set up triggers that implement a wealth of business rules to make sure the new row meets all of the required rules and regulations as dictated by their business and the referential integrity constraints of the database. If the new row does not pass muster, the transaction is rolled back and the change is rejected.

In a similar way, when an UPDATE operation is begun, SQL Server copies the existing row to the DELETED pseudo table and the changed row to the INSERTED pseudo table. As before, the trigger code can validate the change and inspect changed columns to see if they still comply with the rules and constraints currently being enforced.

Note that when you execute an UPDATE or DELETE statement, the operation can span many rows. Because of this, the UPDATED and INSERTED pseudo tables can contain any number of rows. The Trigger needs to be able to deal with all of these rows in succession. Note that when you use the ADO.NET Update method (as exposed on the SqlDataAdapter and the TableAdapter), each UPDATE is submitted individuallyeven in batch mode. ADO.NET makes no attempt to correlate the UPDATE operations. Sure, you can create an ADO.NET SqlCommand object to execute a single UPDATE or DELETE operation that spans many rowsbut you'll have to roll that T-SQL statement yourself.

As shown in Figure 13.104, the INSERTED pseudo table is accessed like any other table in SQL Servervia a SqlDataReader. Note that I must use a context connection to access the list of changes exposed in the INSERTED table.

Figure 13.104. Accessing the INSERTED pseudo table.

[View full size image]

The logic in the second half of the HandleInsert routine (shown in Figure 13.105) walks through the row(s) of the INSERTED table and calculates whether the payment computed by the client application is not only correct, but meets the business rules. In this case, I set the criteria for a "valid" loan at 1100 (monetary units). Yes, this is not the same as the stored procedure, which accepts loans above 1,000 units, but I wanted to use this to illustrate how one tier can accept data and another reject it.

Figure 13.105. Executing the business rule and rolling back the transaction.

[View full size image]

If the logic determines that the loan is unacceptable for any reason, I pick up the current transaction status and roll back the transaction.

To get access to the Transaction class, you'll need to add a reference to System.Transactions.DLL to your project.

Handling the Update

When an UPDATE operation changes the targeted table, your trigger fires with the appropriate TriggerAction set to Update. In this case, the Framework exposes a count of the columns in the UPDATED table, which can, in turn, be used to dimension a Boolean array to test to see which columns have changed. Once this array is populated, you can walk through the values to fire column-specific testsbut just on changed columns (if that makes sense). In this case, I know that column 6 contains the payment, and it's the only one I care about. I perform the same test on this column using the HandleInsert routine.

Figure 13.106. Handling UPDATE operations.

[View full size image]


Advanced Debugging

Yes, you'll find there are times when you want to debug your CLR executable, but you don't want to rebuild and redeploy it beforehand or (more importantly) don't want to execute the test.sql test script. This can be accomplished by manually attaching Visual Studio to the target SQL Server instance hosting your CLR executable. This is not that hard to set up, and Visual Studio 2005 makes it easier than ever. There are a few surprises here that might take you off guard, so read this section carefully before wandering into the swamp.

Suppose you have a SQL Server that's hosting the CLR executable you want to test. In this case, I'm using BETAV9 as the remote system. To debug CLR executables on that system, you'll need to "attach" to the specific process on that machine that's hosting the CLR executable. To set this up, follow these steps:

1.
In the Visual Studio IDE, load the CLR executable project and click "Debug | Attach to Process" from your CLR executable project. This exposes the dialog shown in Figure 13.107.

Figure 13.107. Attaching to a selected SQL Server hosting your CLR executable.

[View full size image]

2.
Check "Show processes from all users" to list processes being run as other users. SQL Server is not typically executed with your user credentials (or shouldn't be).

3.
Scroll down in the dialog to locate the specific instance of SQL Server hosting your CLR executable. Nope, these are not listed by instance name (I'm not sure why not), but by process name and ID. If you're not sure what process to attach to, use the SQL Server Configuration Manager to identify the ProcessID for the specific instance hosing your CLR executable.

Note that the target process must have T-SQL listed as an attachable "type". This is not included in the list if CLR executables have not been enabled on the specific SQL Server instance.

4.
Click "Attach" to begin the attachment process. At this point, you'll see a dialog, shown in Figure 13.108, that warns you about potential harm to your computer. If you feel lucky, click Attach.

Figure 13.108. The Attach Security warning.

5.
At this point, the current instance of Visual Studio 2005 and every other instance of Visual Studio 2005 is now in debug mode. Right, that's what I said. Attaching to a process when you have multiple instances of Visual Studio started switches all of them into debug mode. And no, once you attach to a process for debugging, you can't attach another instance of Visual Studio to that server instance. Strange.

6.
Now that you're in Debug mode, you can set/reset breakpoints and view the state of specific variables, but you can't edit and continue. To test your code, you can execute queries directly against the target SQL Server. If the T-SQL accesses your CLR executable and you've set a breakpoint in the CLR code, the Visual Studio IDE is topped and you'll be in step-through mode.

Since you'll be interactively debugging SQL Server and CLR executables, keep in mind that the test query you're executing could time out and confuse your testing. Use the Tools | Options | Query Execution dialog in SQL Server Management Studio to set the Execution time-out value. It defaults to 0 (infinite timeout), so this should not be a problem unless you've changed it.

It's also important to note that all CLR executables on the instance are paused when you step into your test CLR executable. This means it's especially important to avoid debugging on a production server.

When you're done debugging, simply click the "Stop Debugging" button in the Visual Studio IDE. This detaches the server from your code.


UDT Safety and IP Security

Depending on what your CLR Executable needs to do, you might find that SQL Server and the Framework won't let you deploy it until you alter the Project Permission Level settings. I discuss the nitty-gritty details of code safety in another chapter, but I did want to include a brief description of the types of operations that might prevent your code from running as "Safe"the default setting.

Consider that all of the examples implemented so far in this chapter have been run as Safe. I expect that your DBA will appreciate you sticking to CLR executables that are "safe," but this might limit the functionality you need to implement in your CLR code. The other settings allow your application more freedom in what it can do. However, with that freedom comes more responsibility. Under some circumstances, an "Unsafe" CLR executable can bring down SQL Server and the system itself.

The three possible settings are:

  • Safe(SAFE): This setting permits your CLR executable to perform only internal computation and local data access. Local data access means you can open connections only to the local context connectionnot to an external server. Yes, SAFE is the most restrictive permission set. CLR executables executed by an assembly with SAFE permissions cannot access external system resources such as files, the network, environment variables, or the Registry. In addition, you'll find that a SAFE assembly can't access many of the more sophisticated Framework classes, like encryption.

  • External (EXTERNAL_ACCESS): This setting permits your CLR executable to have access to external system resources such as files, networks, environmental variables, and the Registry. It's required if you need to make a non-context connection.

  • Unsafe (UNSAFE): This setting allows assemblies (almost) unrestricted access to resources, both within and outside SQL Server. Code executing from within an UNSAFE assembly can call unmanaged code. This is clearly the most dangerous of all settings and should be avoided whenever possible. Even UNSAFE prevents access to a number of .NET Framework classes. These include compilers, window managers, enterprise service libraries, and other classes where Microsoft did not think it makes sense for CLR executables to access.

I also want to quote from the documentationjust to be sure you see this warning in Microsoft's own words:

SAFE is the recommended permission setting for assemblies that perform computation and data management tasks without accessing resources outside SQL Server. EXTERNAL_ACCESS is recommended for assemblies that access resources outside SQL Server. EXTERNAL_ACCESS assemblies by default execute as the SQL Server service account. It is possible for EXTERNAL_ACCESS code to explicitly impersonate the caller's Windows Authenticated security context. Since the default is to execute as the SQL Server service account, permission to execute EXTERNAL_ACCESS should only be given to logins trusted to run as the service account. From a security perspective, EXTERNAL_ACCESS and UNSAFE assemblies are identical. However, EXTERNAL_ACCESS assemblies provide various reliability and robustness protections that are not in UNSAFE assemblies. Specifying UNSAFE allows the code in the assembly to perform illegal operations against the SQL Server process space and hence can potentially compromise the robustness and scalability of SQL Server.

To this warning, I would like to add that the designations are really backward. That is, "safe" code is really totally untrusted. Because it's untrusted, it won't be permitted to do anything, access anything, or branch anywhere that would cause damage (other than writing garbage to the database, dropping objects, and doing other harm with the scope of the rights granted the user executing the code. On the other hand "Unsafe" code must be totally trusted. Because it's trusted, it is permitted to do anything, anywhere on your system or any system in sight.

Exposing Your Intellectual Property

By default, Visual Studio 2005 saves the source code for your executable into the system catalogs along with the binary DLL. This means that anyone with access to the deployed database can do something like this (as shown in Figure 13.109):[25]

[25] See ms-help://MS.VSCC.v80/MS.MSDN.v80/MS.VisualStudio.v80.en/WD_ADONET/html/c475eb89-c9e0-4f09-a2a0-bbd237434ea3.htm

Figure 13.109. Extracting the source code for a deployed CLR executable.

Okay, now that the data is returned, it's simply a matter of selecting the source from the returned content, saving it to a .VB file, and opening the .VB (or .CS) file with Visual Studio. There, you will see the entire source filecomments and all. Sigh. This means that database you deployed with your CLR executables in place has also given away your business's intellectual property. Why is this? Well, take a look at the output from a typical build (as shown in Figure 13.110).

Figure 13.110. The result of a typical build.

Notice that the source code (the .vb file), as well as the binary (.dll) and all of the other files, Visual Studio uses to manage your project are written to the database catalogs.

Protecting Your Intellectual Property

So, how do you protect your IP? Well, according to Microsoft, you can't use Visual Studio to deploy your CLR executableyou'll have to build your own deployment scripts. I'm going to leave the details about how to do this to your DBA. Let's just say that you can get started in this process by scripting the deployed assembly and executable, dropping the assembly and all dependencies, and editing the CREATE ASSEMBLY scriptremoving the ADD FILE statements, as shown in Figure 13.111.

Figure 13.111. The unaltered CREATE ASSEMBLY script.

[View full size image]

Sure, the CREATE ASSEMBLY T-SQL function can take the name of your DLL file, so you don't have to type in the binary representation of the executableunless you're being paid by the hour.


Summary

Yes, this is chapter covered a lot of informationit's an important topic for SQL Server 2005 developers and DBAs alike as well as systems architects. Here, we toured an entire suite of examples of T-SQL executables. We saw where some implementations make sense and others fall short. We also saw how Visual Studio can make the process a whole lot easier and faster. Hopefully, these pages can help get you down the road toward your own high-performance CLR executableswithout making your DBA saw your desk in half and throw the splinters into the courtyard.


Chapter 14. Creating and Managing Reports

    Introduction

    Understanding Reporting Services and RDL

    Visual Studio 2005 Reporting

    What Are the Visual Studio 2005 Report Tools?

    Building Your First Report

    Exploring the ReportViewer Class in Depth

    Managing Server Reports

    Managing Parameters

    Advanced Reporting Techniques

    Implementing the Matrix Report

    Summary


Introduction

As many of you already know, Peter Blackburn and I spent over a year writing our previous book, Hitchhiker's Guide to SQL Server 2000 Reporting Services (Addison-Wesley, 2004). This work focuses on reporting technology originally targeted for SQL Server 2005, but when Yukon[1] slipped (and slipped again), the SQL Server team decided to integrate Reporting Services into SQL Server 2000. As a result, Reporting Services has proven to be a very popular new application genrewhat I call "the 5th Paradigm". As I see it, most applications are simply "data viewers," written to capture data with forms and display this data in other forms or summaries and compilations of this data in the form of reports. As far as I'm concerned, any tool that can make this process easier makes sense.

[1] Yukon: A province in northern Canada or the code-name for SQL Server 2005.

Reporting Services 2000 introduced (or leveraged) a Microsoft-owned proprietary standard RDL (XML) report definition language (that's open for other third parties to use), as well as a host of tools, SQL Server databases, an ASP.NET Web Service and a number of Windows services to host, manage, store, distribute, render, and protect the reports and the data they expose. In addition, Reporting Services 2000 includes a Visual Studio 2003based Report Designer that permits developers to author reports using the RAD features for which Visual Studio is famous.

This same Reporting Services toolset was leveraged to create the new Visual Studio 2005 ReportViewer control and Report Designerit closely resembles the Visual Studio 2003 version's Report Designer layout and preview panes. No, this chapter is not really about Reporting Services, but I think that if you understand the parentage of a body of code, you'll better understand its strengths, its weaknesses, and what's been improved (or lost) in the transition. While I'll give you a rudimentary overview of SQL Server Reporting Services in this chapter, I think that the 2005 version of Reporting Services is similar enough to the 2000 version that you should use our previous book as an in-depth reference.

Visual Studio 2005 takes up where Visual Studio 2003 left offsort of. Visual Studio 2005 no longer requires a separate after-market Reporting Services add-in to support SQL Server 2005 Reporting ServicesMicrosoft shipped an entirely separate instance of Visual Studio to host the Report Designer. This new SKU, called the Business Intelligence Development Studio (BIDS), is essentially the Visual Studio 2005 UI with the Report Designer built in. Does this mean that unless you have a SQL Server SKU, you won't be able to build RDL-based reports with Visual Studio? Ah, no. That's where the ReportViewer control comes into play.

That's what this chapter is abouthow to create new RDL reports in Visual Studio 2005, how to use the ReportViewer control to render them, and how to access RDL reports hosted by SQL Server Reporting Services. Since many of you have invested considerably in Reporting Services 2000 technology and created literally millions of RDL reports, I know that many of you want to know how to import the RDL report definitions into your Visual Studio 2005 report projectsI'll show you how to do that, too. So let's get to it.


Understanding Reporting Services and RDL

As I said, the core reference on SQL Server Reporting Services is our book on Reporting Services[2], but to summarize a 784-page book, Reporting Services is an ASP.NET service that manages a SQL Server database that maintains a database repository of reports formatted using the industry-standard[3] XML-based report definition language dubbed "RDL." An RDL report might contain references to the actual source of data, but this is not a firm requirement. It does include references to the data "columns" or fields to be displayed or used in expressions managed by the RDL report generator. SQL Server Reporting Services maintains these RDL definitions in a SQL Server database for on-demand retrieval and rendering. Based on parameters passed to the ASP.NET Reporting Services service, the report is connected with a data source, the data is retrieved, and the report is constructed and rendered given the format and expressions built into the report. RDL definitions can also incorporate values from input parameters used to focus the query, change the way a report is rendered, or deal with any number of other issues, including formatting, special rendering or printing instructions, and many others.

[2] Hitchhiker's Guide to SQL Server 2000 Reporting Services (Addison-Wesley, 2004; ISBN 0321268288).

[3] W3C standard RDL.

Figure 14.1 shows a broad overview of the SQL Server Reporting Services architecture. Basically, Reporting Services is an IIS-based executable that exposes its functionality via Simple Object Application Protocol (SOAP)[4]. The Visual Studio Report Designer is used to create the reports and deploy them to the SQL Server Reporting Services reports database catalog. The Reporting Services engine has total responsibility for the reportit manages the connection, executes the query (including prompting for and managing query and report parameters), generates the report based on the RDL report definition, and renders the report in one of a number of user-selectable formats. Reporting Services also includes mechanisms to protect reports and the data they expose, and permit reports to be automatically distributed on demand or on a predetermined schedule. The Reporting Services toolset includes the Report Manager, which is a browser-based tool used by DBAs and users alike to configure, tune, and access reports exposed in a hierarchical directory structure hosted on the IIS and SQL Server.

[4] www.w3.org/TR/soap/

Figure 14.1. SQL Server Reporting Services overview.

[View full size image]

To give you an idea what I'm talking about when it comes to RDL-formatted XML, take a look at Figure 14.2it's an RDL file exposed in a browser as XML.

Figure 14.2. A typical report's RDL file displayed as XML.

[View full size image]

The RDL in this specific report was generated using the Visual Studio Business Intelligence Development Studio (BIDS) Report Designerit contains the following elements that describe the report:

  • This RDL report definition addresses a data source whose actual connection details are kept elsewhere indexed by a GUID. This means that this report must be "hooked up" with a specific Data Source that contains the actual ConnectionString details when it's time to render the report. Yes, an RDL definition could point to a "private" or report-specific Data Source, and in this case, the RDL contains <ConnectionString> and <DataProvider> elements.

  • The RDL report definition describes the report size, shape, and margins, along with other formatting details, but does not specify the manner in which it should be rendered. That is, while the report is described in every detail, there is nothing in the RDL that says how it should be displayed in a browser, on a printer, or in an Adobe PDF file.

  • Some of the elements in the RDL file (like <ReportID>) are not part of the RDL specification. If you look closely, you'll see that ReportID is prepended with rd:, which means it's part of the rd namespace. If you look at the top of the document, you'll see that the rd namespace is that for the Report Designer. There are lots of undocumented parts of the RDL that you'll see as part of the rd namespace that are used by the Report Designer. It is possible to extend RDL to utilize your own namespaces as long as you properly reference them.

  • The report body section contains elements and attributes (XML-speak) that specify the layout of the report items such as TextBox, Table, Chart, and such. These are collapsed behind the +<Body> tag. The RDL in this section describes how each item is laid out and how they are to be populated with data or expressions.

  • The DataSets section contains the Field definitions for each dataset used in the report, along with a Query definition used by the data processing extension to retrieve data.

  • The DataSets section contains the SQL query that returns columns that map to the data fields to be mapped to specific report elements. Yes, field values can be expression-based. However, it's not possible to combine fields or create expressions that combine fields from different datasets in a report.

The RDL shown previously was used to create the report shown in Figure 14.3 [5].

[5] Well, yes, the RDL shown above is only a subset of the entire RDL report definition.

Figure 14.3. A simple sample summary report.

[View full size image]

This report was built using the BIDS Report Designer in about 10 minutes. Okay, it might take someone unfamiliar with the Report Designer a bit longer, but the process is really pretty easy. As I said, this is not a chapter about the BIDS Report Designerit's about Visual Studio 2005 and how it implements this technology. However, whether or not you've used Reporting Services, you might (just might) miss some of the features supported by BIDS that are not supported in Visual Studio. The following list summarizes what you'll have to do withoutbasically, Reporting Services supports the following features not exposed in Visual Studio 2005:

  • The BIDS Report Designer includes a report wizard that walks you through the process of building reports or a new report project. The Report Project Wizard walks through the process of setting up a shared or report-specific Data Source and the initial report query.

  • The BIDS report wizard walks you through the process of creating a report from the initial query. It also leverages several (fixed) layout templates that can be used to customize the look of your reports. It can help get you started with either "table-based" or "Matrix" reports, build templated headings, and help build basic groups.

  • The BIDS Report Designer includes interactive Data Source and DataSet designers that support easy WYSIWYG creation or alteration of the queries used to populate reports and parameter pick lists. This interface includes both a generic query interface that can take virtually any query syntax, including stored procedure invocation, and a "GUI" query designer used to create simpler table-based queries.

  • The BIDS Report Designer supports several different techniques to implement and configure "code-behind" routines that can be invoked directly from the query designer or behind the scenes in the report properties.

  • The BIDS Report Designer includes an interactive report parameters page that permits developers to specify a variety of ways to configure how parameters are defined, set to default values, or populated from pick lists. This feature should not be under-estimated in its power and usability.

  • The BIDS Report Designer is extensible with third-party controls like the new add-in Chart control from Dundas[6]. Don't expect to be able to use these controls in the Visual Studio 2005 Report Designernot anytime soon.

    [6] Dundas is a well-respected third-party add-in vendorand it's pronounced with an Irish accent: "Duundis".

  • Once the report is deployed on the SQL Server report server, it can be called up using the Report Manager ASP page that permits developers, DBAs, and users to access reports, specify and fine-tune security personas, manage (and delete) shared data sources, define subscription schemes, and move reports between directories.

  • Access to the reports from Reporting Services can be channeled through an SSL connection to encrypt all traffic being sent to and from the user. This protects the confidentiality of the report. This approach also means that the user application does not have access to the RDL used to specify the Data Source, credentials, or queries used to generate the report.

  • Report access is governed by a Reporting Servicesspecific user/group/administrator hierarchy of rights that permits the DBA to ensure that only specific users or groups can perform administrative operations or access specific reports or report directories.

  • Reports can be programmed using the Report Manager (report "explorer") to prompt users for report and query parameters or simply run with default parameter settings. These parameters can be managed by the Report Manager (at least, to a great extent), where default values and parameters can be set or hidden on a user-by-user basis (based on the user's group). The report UI supports sophisticated cascading or multi-select parameter schemes.

  • Reports launched from SQL Server Reporting Services can be re-rendered to an XML, CSV (comma delimited), TIFF, or Excel file or a Web Archive format. Users can also choose to tune page setup specifications, hide/display report parameters, and interactively zoom the report view as desired. In the 2005 version, users can choose to route reports to a selected printer.

  • The report server can be programmed using the Report Manager to launch reports at scheduled times, cache reports so they don't have to be re-executed or re-rendered more than once, push reports to specific users, or permit reports to be pulled by users on demand.

Yes, I could go onthis isn't anywhere near the entire list of SQL Server Reporting Services features, but you get the idea. Just understand that Reporting Services is a more sophisticated report-management system designed for companies that have a significant report workload. As Peter has pointed out on many occasions, there is still a lot of work to do to make SQL Server Reporting Services an "enterprise-ready" reporting tool.


Visual Studio 2005 Reporting

Microsoft's Visual Studio team realized that the developer community wanted a better alternative to the complexity of Crystal Report and something more sophisticated than the Visual Basic 6.0 report control paradigm. Visual Studio's 2005 ReportViewer is the result. Basically, the ReportViewer control comes in two flavors: Windows Forms and ASP.NET. The control itself is essentially the same control developers used in Visual Studio 2003 when they previewed reports in the Report Designer and the same control used when the SQL Server Reporting Services Report Manager renders a report. As shown in the architectural diagram in Figure 14.4, the ReportViewer control encompasses many of the features of the Reporting Services server-side engine directly on the client. It includes a "report processor" that interprets the RDLC report definition, processes the report parameters, and renders the report to a variety of user-selectable formats. It's capable of running in "local mode," where, unlike the server-based Reporting Services paradigm, the ReportViewer does not take on the task of managing connections or running parameter-based queries, nor does it prompt for query or report parameters. In "remote" mode, the ReportViewer merely hosts the report-centric Parameters collection, addresses a remote Reporting Services report, and renders itincluding any parameter prompts as defined by the server-side RDL report definition. Neither mode takes on the responsibility of handling the report querythat's up to your code in the local mode or the data processing extension in remote mode.

Figure 14.4. Visual Studio's ReportViewer control in "Local" mode.

[View full size image]

As you'll discover later in this chapter, parameter management is of critical importance. I'll discuss query, local report, and remote report parameter management, as well as how and when to prompt for parameters, pass them to the ReportViewer, and use parameters to focus, tune, and customize your reports.

While the Visual Studio development tools and report support tools are far fewer, thinner, and cheaper, there is nothing to stop you from implementing your own "report server" and providing much of this "missing" functionality (or, at least, the parts you need) using your own code. Something else you don't need when building Visual Studio reports: SQL Server. Nope, you don't need any version of SQL Server to fully utilize the Visual Studio 2005 reporting tools, designers, and ReportViewer control. You'll also discover that the tools I've already discussed that are used to build strongly typed TableAdapters can be used to generate report data. And yes, the ADO.NET code you've been working with for years can also be used to generate untyped DataTable instances to be passed to the ReportViewer.

Virtually all versions of SQL Server 2005 include Reporting Services, including SQL Express. The only version that doesn't support it is the new SQL Everywhere SKU. This means you might want to leverage the more interactive report-authoring tools afforded by the BIDS SKUs.

Yes, you can use the ReportViewer control to reference and render SQL Server 2005 Reporting Services deployed reports, and in some cases, this makes a lot of sense. Sometimes, you need to integrate a report into an application and, after capturing the parameters, simply request the parameterized report from Reporting Services. That's something that the BIDS tools don't really help you with. When users access Reporting Services reports via a browser, they are free to change the URL to alter the parameters or choose other reports or report rendering options, and cause other mischief (subject to the limitations of the security settings). This means you might have to create an application specifically designed to capture parameters and execute reportsthat's right up Visual Studio 2005's report tool's alley.

Tip

You'll need to convert all Reporting Services 2000 Reports before using them with Visual Studio 2005.

While you might miss the BIDS report project, connection, dataset, and design wizards that spit out templated reports after a few minutes of drag-and-drop development work, you'll also find that you quickly outgrow the wizards. You'll soon learn to build your own customized report templates based on your company's report format guidelines. Given how easy it is to add templates to Visual Studio 2005, this should not be a big issue. In addition, I'll show you how to leverage reports developed with any RDL tool (including the BIDS Report Designer) and launch them from your custom Windows Forms or ASP.NET application.

Before I get started, let's touch on a quick technical point: The RDL files created by the Visual Studio 2005 Report template and populated and managed by the Visual Studio 2005 Report Designer generate an RDLC extension file. Yes, this is a variation on the XML RDL-formatted files used in Reporting Servicesthe same, but different enough that they (sadly) can't just be interchanged. One would think that a future version would just deal with the differencesat least, with an import wizard. I'll step through the process of passing these files (with some modification) among Visual Studio 2005, BIDS projects, and SQL Server 2005 Reporting Services.


What Are the Visual Studio 2005 Report Tools?

Visual Studio 2005 leverages the tools, designers, and RDL tuning created for the first SQL Server 2000 Reporting Services tools suite that was hosted by Visual Studio 2003. The new tools include:

  • The Report Designer: This tool is launched when you navigate to a report (RDLC) file in your project. It's basically the "layout" tab in the Reporting Services 2000 Report Designer. No, this designer does not have a "preview" or "data" tab. Those aspects of report development are handled differently.

  • The Report Items toolbox: This includes the same toolbox items as the BIDS Report Designer. These items include a TextBox (label), Line, Table, Matrix, Rectangle, List, Image, Subreport, and Chart. Yes, these items are programmed in the same way as the BIDS Report Designer toolbox items. No, this toolbox is not extensible by third-party control vendors like the BIDS toolbox because the controls are implemented to execute on the Reporting Services server, not on the client.

  • The ReportViewer control: The ReportViewer is used to host and render the RDLC report in a Windows Form or on an ASP.NET page at runtime, or render an RDL report (in "remote" mode). Basically, this control leverages the development Microsoft did creating the BIDS Report Designer's "preview" window and the Report Manager's rendering window. Its job is to read through the data provided by the report query, apply any report parameters, and interpret the RDLC file to render the report in the space provided on the Windows Form or ASP.NET page using Windows GDI or HTML. It renders any report headers, footers, and grouping, and executes any expressions or code-behind logic. It also maintains totals, subtotals, and data groupings, including managing data and page breaks as neededwell, at least, it tries to handle page breaks. I'm of the opinion that this code is "broken." You'll find that you spend an inordinate amount of time reworking layouts as you move from one rendering format to another.

As you can see, the Visual Studio 2005 report tools are a lot fewer than the BIDS tools. You'll be responsible for handling all data access work and a lot more, as you'll soon see. This means you'll need to write code and build UI elements to:

  • Capture local report and query parameters, including all data queries or other logic to populate parameter pick lists

  • Build and instantiate the data access interfaces, Data Sources, and classes

  • Open and close the connections as needed

  • Run the report and pick-list queries in the sequences that make sense

  • Pass the rowset(s) to the ReportViewer

  • Get the ReportViewer to render the report

Yes, virtually all of these operations (and more) are executed automatically behind the scenes when you launch a BIDS report.


Building Your First Report

Sometimes, it's best if you just get a quick drive-by look at someplace new before getting out and walking through all of the buildings, streets, back alleys, and sewers. To that end, let's walk through the process of creating a sample local[7] report using the new Visual Studio 2005 report tools. Nope, you don't need SQL Server (any version) to get this to workbut it won't hurt. You will need a source of data, though. It could be almost any data sourcea flat file, a business data object, or a database tableeven if it comes from Oracle. In this case, I'll use something exotic: the SQL Server sample "AdventureWorks"[8] database.

[7] The ReportViewer control can host either local (client) RDLC reports or address remote (server-hosted) RDL reports.

[8] This database is on the DVD and is also available as a sample database installed with SQL Server 2000 and 2005. The version on the DVD has a number of additional stored procedures referenced in some of the reports.

To set up the sample report application:

1. | Start Visual Studio 2005.
2. | Create a new project. I'm going to stick with a Windows Forms application, but this works about the same in an ASP application. I'm using Visual Basic .NET and have named the initial project prjSampleVSReport.
3. | Your project starts with a blank form (as expected) and a nearly empty project explorer.
4. | Right-click on the project (prjSampleVSReport, in this case) and choose "Add | New Item". Choose "Report" from the Add New Item "installed templates" list. A bit later, I'll show you how to add your own custom reports to this list. Before you click Add, name your report's .RDLC file. I'm going to call this report "ProductByProfit", as shown in Figure 14.5.

Figure 14.5. Adding a new report to the project.

[View full size image]

Creating a Report Data Source

In the following few steps, I'll use the DSCW to create a Data Source I can use to populate the report items. I need to build and configure the report before I attempt to "bind" the report to the Form or ASP pageif I don't, Visual Studio won't help us by generating the needed TableAdapter fill method.

1.
I don't have a report wizard to prompt us to set up a source for the report data, so I need to add that to the project one way or another. As you'll see later, there are a number of ways to retrieve the rowset used to populate the report. For this example, I'm going to use the new Visual Studio 2005 Data Source Wizard.

2.
Click on the Data menu and click "Add New Data Source". Yes, this is the same technique I discussed in Chapter 7, "Managing Data Tools and Data Binding," so I won't be showing much more detail here.

3.
For this example, I used the Data Source Configuration Wizard (DSCW) to access a Database table in the AdventureWorks database (the sample database that ships with SQL Server).

4.
Choose the Product (Production) table (under Tables), but choose only a few of the columns: Name, Color, ListPrice, and StandardCost. Before you leave this dialog, I like to rename the DataSet name to a shorter nameit just makes it easier on everyone. I called the DataSet "AWDataSet", as shown in Figure 14.6.

Figure 14.6. Choosing columns from the Products(Production) table.

[View full size image]

To "deselect" all of the columns, click on the "Table" icon again. This deselects the table and the selected columns. Next, select just the columns needed.

5.
Click "Finish" when you're finished choosing columns for the report rowset.

Yes, this interface is kinda crippled, as I generally return report rowsets from a server-side JOIN and not just a single table. There are a couple of easy solutions to this problem: Create a View or stored procedure that returns the right SQL project. Remember, later on in this chapter, I'm going to show you how to write your own code to generate report rowsetsincluding those that require a parameter or two to focus the WHERE clause on fewer rows. As it is, this query returns all of the Products (Production) tablenot good.

I still need to make a change to the generated TableAdapter so it returns the right values for this report. I know you can create a table-dump query (as I've just done), but there are very few situations where this makes sense, so let's step into the TableAdapter and tune up the Fill, so it returns the data I need to start withat least, to a limited extent.

6.
From the Project Explorer, choose the AWDataSet.xsd file and double-click to open it. As I discussed (several times) in Chapter 7, I need to customize the query so it returns an expression from the data I've fetched.

7.
Right-click on the "SQL" icon in the Product table property dialog in the TableAdapter XSD designer. Navigate to the query builder and change the SQL to include an expression to return Profit (ListPrice-StandardCost), as shown in Figure 14.7. Click OK in the Query Builder and Finish to complete the changes to the TableAdapter. Behind the scenes, Visual Studio regenerates the strongly typed TableAdapter to incorporate these changes.

Figure 14.7. The altered SQL query to return and sort by the "profit" expression.

[View full size image]

At this point, the Data Sources window should show the new expression-based column, as shown in Figure 14.8. Note that the process of adding a new column could be handled directly by the BIDS Data dialog without changing the report row source query.

Figure 14.8. The Solution's Project Explorer showing the new Data Source.

Laying Out the Report

I'm now ready to lay out the report and program the RDLC. No, that does not mean you need to learn XML and RDL so that you can enter the arguments directly, unless you really want to. This process is very much like any other report or RAD development tool you've ever used, but it's generally easierthere are far fewer "controls" to choose from.

1.
Click on the ProductByProfit.rdlc report work surface and lay out the report elements. I'll need a TextBox and a Table report item. Double-click or drag-and-drop these items from the Report Items toolbox (as shown in Figure 14.9) to the report design surface.

Figure 14.9. The Report Items toolbox.

2.
Align the TextBox and Table items as shown in Figure 14.10.

Figure 14.10. Add a TextBox and Table item to the report design surface.

[View full size image]

3.
Select the TextBox item and enter the title of the report. Yes, I use TextBox report items to display headings and other information that's not (generally) data-driven. Sure, you can write an expression that feeds any report item in a variety of ways. In this case, I set the Text property of the TextBox report item placed at the top of the report (the report heading) to "Products By Profit".

4.
Use the Report Formatting toolbar (shown in Figure 14.11) to decorate and embellish the TextBox font. In this case, I centered and set the text and background color. The properties for each report item can be set using the Visual Studio IDE's property setter dialog, via right-click custom property pages or via the report-centric toolbars.

Figure 14.11. Setting attributes with the Report Formatting toolbar.

[View full size image]

5.
Next, I'll populate the Table report item. The Table report item is described in infinite detail in our Reporting Services book. Suffice it to say that the Table report item is a data repeater that's designed to display data from each row in the report rowset and provide a detail row (at least) and (optionally) a header and footer row that's populated once before the first row and after the last row. Sure, it's possible to set up multiple groups of data, as I'll illustrate (briefly) later.

IMHO

Remember, the best source of in-depth report layout and design information can be found in our Reporting Services book.

6.
Since we'll need an extra column for the Profit values, right-click on the last column header (the top colored border) in the Table report item and choose "Insert Column to the right".

7.
Resize the columns so they fit under the heading TextBox report item. Don't worry about being too precise about alignment at this point. You'll undoubtedly need to resize the columns later once you get a look at the first report prototype.

8.
Open the Data Sources window so it can be used to drag elements to the Table report item. Drag the Product Name icon to the first column, middle section of the Table report item. The Table report item should fill in the column name into the Header cell (top row) and an expression to reference the Name column in the middle (Detail) cell, as shown in Figure 14.12.

Figure 14.12. Populating the Table report item from the Data Source columns list.

9.
Repeat this process for the ProductNumber, Color, and Profit columns. Your report layout should look something like Figure 14.13.

Figure 14.13. The report layout after all columns are placed into the Table item.

[View full size image]

10.
Yes, you can embellish the column formatting at this point. You'll probably want to set the formatting for the Profit value cell to "C", center the column headings, or perhaps decorate the header row fonts. Of course, this assumes that the report will always run on a computer where the installed regional currency is the same or where you've set the appropriate language properties of the report item. If this is not the case, you need to set the language property of the report TextBox item. To do so, select the cell or entire row and navigate to the properties page or one of the report-formatting toolbars and select the appropriate options.

Nope, you're not ready to "run" the report yet. You have not hooked up the ReportViewerthat comes next.

Configuring the ReportViewer Control

At this point, it's time to add the ReportViewer control to the Windows Form. Remember, this control interprets the RDLC file I just configured using the Report Designer "layout." It also expects the Data Source to be fully populated, so you need to ensure that the TableAdapter Fill method is executed somewherehopefully before you try to launch the report.

1.
Select the Windows Form tab in your project (Form1.vb [Design]). This exposes the Windows Form1 work surface and the Windows Forms Toolbox. Navigate to the Data section of the toolbox and double-click on the ReportViewer control. This installs the ReportViewer control on the Windows Form, as shown in Figure 14.14. Note that the default ReportViewer is too large to fit the form, so you'll have to make either one larger or the other smaller.

Figure 14.14. Adding the ReportViewer control to the Windows Form.

[View full size image]

The ReportViewer control is the host for any report your application will expose to the user. This could be the RDLC report I'm building or an RDL report hosted on a Reporting Services server. Generally, this means the ReportViewer needs to be large enough to view the entire report workspace. You might try docking the ReportViewer in the parent Form, but that opens another can of worms. Let's not go there yet.

When you first access the ReportViewer control, the ReportViewer Tasks pane is opened. This gives you an opportunity to choose the report the ReportViewer control is to host. For this example, I'll simply point to the prjSampleVSReport.ProductsByProfit.rdlc RDLC report I already added to the project. As I discuss later in this chapter, you can also use the ReportViewer to address "server" reportsthose hosted on a Reporting Services 2005 SQL Server.

2.
Once you choose a report, the remaining dialog items in the ReportViewer Tasks are populated, as shown in Figure 14.15. In this case, I don't need to touch any of theseI recommend that you don't.

Figure 14.15. Choosing the report to host.

If you try to choose a report (an RDLC file) that does not have any report items, Visual Studio can't figure out what query needs to be executed to populate the report rowset.

3.
Next, I need to verify that the report Data Source is properly wired. By default, when you choose a report (an RDLC file) that has been populated with report items (as ours has), the Visual Studio report interface inspects the TableAdapter and determines which method needs to be used to populate the report rowset. It then adds a call to the TableAdapter Fill method to the Form1_Load method along with a RefreshReport method invocation, as shown in Figure 14.16.

Figure 14.16. The ReportViewer inserts a Fill method call into Form1_Load.

[View full size image]

No, you're probably not going to want to leave this Fill method invocation or the Me.ReportViewer1.RefreshReport call here in Form1_Load. That's because it will (at least) slow down getting the program's initial form loaded and displayed for the first time. You'll probably want to move this generated code to a Button_Click eventbut you'll do that later. You might want to execute the query that constructs the DataTable in another thread to help performance, but that's fodder for another chapter.

Testing the Report

Nope, there is no "Preview" button on the Report Designer as there is in Reporting Services. This means the only way you can preview your report is to run the application that hosts the ReportViewer control. When you do, the code inserted into your project when binding the selected report to the ReportViewer control executes the Fill method and the RefreshReport method that renders the report. Try itclick on the debug "Run" button or press F5. You should now see something like Figure 14.17.

Figure 14.17. The first test run for the Products by Profit report.

Polishing the Report

No, this is not exactly a finished report suitable to show your bossunless you're looking for a new job. Let's step through a number of steps to make it closer to something that your boss might expect of a professional report developer.

1.
Return to the Form1.vb [Design] window and select the ReportViewer control. In the upper-right corner there is a tiny arrowclick on it to open the ReportViewer Tasks dialog. In this dialog, click "Dock in parent container". This permits the report to stretch with the Form as the user makes it larger.

Next, I want the products with the highest profit to be shown in the report first instead of last. This can be accomplished in a couple of ways. One approach is to change the initial query to sort the data on the Profit expression. This puts a bit more load on the data source server but means that the client does not have to do the sort. Of course, the data source server might run the sort more efficiently, as it can leverage existing indexes. On the other hand, once the subset rowset arrives, it's pretty easy for the client to sort (or resort) the rowsetespecially if there are very few rows to sort. Let's sort the data here on the clientright in the report definition. For an interactive report that spans only a few pages, this makes a lot more sense performance-wise. As the rowset increases, it's going to be increasingly more expensive to sort the data, regardless of whether it's done on the client or the server.

2.
Return to the ProductByProfit.rdlc [Design] window (the report layout design surface). Here is where you can change the sort order of one or more report columns and set other expressions to make the report display a more professional appearance. Yes, these sort specifications are persisted in the RDLC report definition.

3.
Click-select the Table report item and right-click to open the Table Properties dialog, shown in Figure 14.18. Note that I've also filled in the Tooltip property. This and the other Table properties can also be set via the traditional Visual Studio Property page, but this dialog makes setting the Table report item properties easier.

Figure 14.18. The Table report item Properties page.

4.
Since I want to change how the rows in the Table Report item are sorted, click on the Sorting tab, which exposes a dialog as shown in Figure 14.19. The Sorting expressions listed here dictate how the data presented in the Table report item is ordered. You can have as many sort expressions as make sense. As I said, these sort operations are carried out by the ReportViewer on the client systemnot by the data source query engine. Choose the column(s) on which to sort the report data rowset. For this example, I chose the Fields!Profit.Value column. I also set the sort Direction property to Descending.

Figure 14.19. Choosing a field for the sort expression.

Note that the expression syntax for the report rowset (exposed by the TableAdapter, in this case) references an individual column as a "field." The DataTable columns are mapped to this Fields collection. The "bang" syntax (which uses ! to delimitate the Fields collection from its members) is a throwback to early Visual Basic DAO Field column syntax. Note that I'm referring to the Value property of the specific Field. In most cases, if you use the expression editor (which I'll show you many times later in this chapter), the Report expression editor UI generates the correct syntax for each column you need to reference.

Setting the Color Column Background Property

To make it easier for the report user to visualize the product color, you might want to set the background color of the "color" cell to a matching color. Okay, this is a bit silly, but it's an easy way to illustrate how to set expressions on specific report item properties. Keep in mind that these customizations often don't "cross-render" very well. That is, they might look cool in a Windows Forms application, but when rendered in PDF, they look like the stuff left on the carpet by your new puppy after she's eaten the cat food (again).

1.
Select the Table report item "color" cell and press F4 to open the Properties dialog for this cell. Notice that the "cell" is exposed as a TextBox report item hosted in the Table report item. Navigate to the BackgroundColor property. The default setting is "Transparent", which simply means "no color" (kinda like "null", but not really). As discussed in depth in our Reporting Services book, "Transparent" means that the BackgroundColor tag is not generated and the color of the item takes on the color of what's behind it.

2.
I'm going to set this property using an expression, so click on the property and choose "<Expression...>" from the list of available colors. This opens the Edit Expression dialog shown in Figure 14.20.

Figure 14.20. Editing the BackgroundColor property expression.

[View full size image]

Using the Expression Editor

The Edit Expression dialog exposes an easy way to construct expressions for most of the report item properties. As your report-development skills get honed, you'll visit this dialog quite frequently. In this case, I replaced the existing expression ("Transparent") with a reference to the Color Field returned by the query. Because the cell whose property I'm editing is in the "details" row of the Table report item, the ReportViewer control interprets these expressions at runtime on a row-by-row basis. Other property expressions can be applied to the entire report, just the header, just the footer, or other specific elements.

Think of the expression as the value in an assignment statement such as:

TextBox. BackgroundColor=Fields!Color.Value

I encourage you to explore the other constants, Globals, Parameters, DataSets, Operators, and Common Functions. This will help you understand what can be selected as component building blocks to build your expression. Sure, the expression can be fairly complexbut be careful. Although the code is syntax checked, it's not "precompiled," so reference errors won't show up until runtime, when the report needs to interpret the expression and actually access the elements.

If you jump the gun and try to test the report at this point, you'll immediately discover that you're not done polishing. Consider what happens when you use a black font on a black backgroundit's kinda black. Because of this, you need to add another expression to the cell Color property to set the font color to white when the color is "dark," like black or blue. I'm going to let you figure out how to get to the right property expression, but I'll give you a hint as to the expression in Figure 14.21.

Figure 14.21. Setting the Color property expression.

The IIF construct is widely used in report expressions but can be problematic if you aren't careful. It simply returns the first value if the expression is True and the second if the expression is False. In this case, I've doubled up the expression, so it reads "If the column Color data value is black, use white (as the font color), else if the column Color is blue, use whiteif it's neither, use black". Make sense? I discuss how to construct complex expressions in the Reporting Services bookincluding how to build "code-behind" modules that can return values using far more complex code.

It's entirely possible to make changes to the RDLC report definition via the report layout designer while the application is being executed. No, these changes are not applied until the ReportViewer recompiles the RDLC when first executed.

Buffing the First Coat of Polish and Retesting

Before testing the report, I made a few more changes to further polish the report. You have enough instruction at this point to make these changes yourself:

  • Set the background color of the Table report item header row to an attractive color.

  • Set the header row TextBox and Color cell font bold property to True.

  • Resize the initial Form so it's large enough to display the report.

  • Resize the Table report item columns to more closely match the size of the data returned.

Once these changes are made, re-run your application and preview the new report. If you've been following along, your report should look like Figure 14.22. This report looks like it's about ready to send to your boss. No, this process did not take that long, so you might want to pretend you're working hard on making the report right before turning it inif the boss thinks the process is too easy, he might think he can do it himself.

Note that this report has been "zoomed" to 75% by selecting the zoom level drop-down at runtime. This option permits you to see a report rendered based on the size of the target page width, the whole page, or stepped zoom levels from 500% to 25%.

Figure 14.22. The polished report zoomed to 75%.

[View full size image]


Exploring the ReportViewer Class in Depth

When creating simple reports, you won't usually hard-code the properties or call many of the methods exposed by the ReportViewer, with the obvious exception of the ReportViewer.RefreshReport method. However, as your reports and report applications get more sophisticated, you'll want to leverage the underlying classes used to manage reports. For example, if you want to use a menu to launch a selected report, you can use the ReportViewer instance's LocalReport property (which exposes a LocalReport instance) to point to a specific RDLC file and its associated Data Source. This way, you can reuse a single ReportViewer instance or instantiate a new form that hosts the ReportViewer. I'll show an example of that application later in this section. Another approach might be to permit your application to launch any selected report from its source RDLC file. In this case, you would have to extract the embedded queries and create an appropriate Data Source or provide one through other means. That application is fodder for a whitepaperany volunteers?

The ReportViewer Properties

Let's take a short look at the fundamental ReportViewer properties, methods, and events. I'll save a detailed look for later. All of these (basic) properties are exposed via the Visual Studio property page:

  • CurrentPage: Gets or sets the report's current page. This can be used to get the ReportViewer to navigate to a specific page in the report.

  • DocumentMapCollapsed, DocumentMapWidth, and ShowDocumentMapButton: These properties determine how the Document Map and the button to activate a Document Map are displayed. I illustrate programming and use of Document Maps in our Reporting Services book and in the final section in this chapter. Basically, this is an area exposed on the left side of the report that permits users to navigate to a specific section of the report. For example, the report shown in Figure 14.23 includes a document map that permits easy navigation to each section of the reportone section for each product color.

    Figure 14.23. Enabling the Document Map.

    [View full size image]

  • HyperlinkTarget: Gets or sets where to go when a hyperlink in the report is clicked. This link can point to a subreport or to a URL.

  • LocalReport: Addresses and describes a LocalReport instance that points to the local RDLC report (file) and Data Source being managed by the ReportViewer. I discuss the LocalReport class in depth later in this section and throughout the chapter.

  • ProcessingMode: Determines whether the ReportViewer is processing a client-side (local) or server-side (remote) report. This property defaults to "local".

  • PromptAreaCollapsed: When linking to remote Reporting Services reports, this property determines whether the parameter prompts are shown (the default) or hidden. This property setting has no effect on "local" mode reportsit makes sense only for remote mode.

  • ServerReport: This property addresses and describes a ServerReport instance. When set, the ReportViewer is set to "remote" mode. The ServerReport instance addresses a Reporting Services 2005 deployed report. I discuss the ServerReport class later in this section.

  • ShowCredentialPrompts: This property (used only in remote mode in server-hosted reports) determines whether the user is prompted for logon credentials as specified by the Reporting Services deployed report configuration.

  • ShowExportControls, ShowFindControls, ShowPageNavigationControls, ShowPrintButton, ShowPromptAreaButton, ShowToolbar, ShowZoomControl, and ShowRefreshButton: These properties determine which option buttons are displayed by the ReportViewer toolbar, as shown in Figure 14.24. Depending on what functionality you want to expose to your application user, you might want to hide some of these buttons (which are all on by default).

    Figure 14.24. The ReportViewer toolbar.

    [View full size image]

  • ShowReportBody: Determines whether the report body should be visible in the ReportViewer.

  • ZoomMode and ZoomPercent: Gets or sets if the report is shown 100% or zoomed in or out and at what percent.

The ReportViewer Methods

The following methods are those you're most likely to access.

  • JumpToBookmark, JumpToDocumentMapID: Navigates the ReportViewer to the selected bookmark or document ID.

  • PerformBack: Navigates the ReportViewer back to the parent report from a drillthrough report.

  • RefreshReport: Calling this method flushes the current report and re-renders the report using the current state of the DataTable bound to the LocalReport or ServerReport property. A call to this method is added to Form_Load whenever you make a change to the ReportViewer Data Source via the ReportViewer Tasks dialog.

  • Reset: This method clears the ReportViewer properties to their default state and replaces the LocalReport object with a new instance. This method might be necessary when you set up a report with the ReportViewer UI but want to override those settings at runtime. This method is not documented but is accessible in the class.

The ReportViewer Events

The following events are those you're most likely to use to monitor your report application:

  • Back: This event fires when the user navigates back from a drillthrough report.

  • BookmarkNavigation: Fires when the user navigates to a bookmark in report.

  • Drillthrough: Fires when a report item programmed with an action is clicked. This event is responsible for populating the targeted report's Data Source. I show an example of its use later in this chapter.

  • DocumentMapNavigation: Fires when the document map is selected.

  • Hyperlink: Fires when the user clicks a report hyperlink.

  • PageNavigation: Fires when the user changes the current page.

  • Print, ReportExport: Fires when the user requests to print or export the report.

  • RenderingBegin, RenderingComplete: Fires as the report begins and ends rendering.

  • ReportError: Fires when an error occurs in the report. This event returns a ReportError instance that contains the Exception and Handled arguments. Handled is True if the host application has dealt with the exception.

The LocalReport Class

The ReportViewer Tasks dialog is your UI portal to the LocalReport (or ServerReport) instanceor, at least, some of their properties. If you set the ReportViewer ProcessingMode to Local (or don't change the default setting), the LocalReport instance contains properties to address the report being displayed, address the Data Source(s), and set other properties to describe the report and how it behaves. Only a few of these properties are exposed via the ReportViewer Tasks, so you'll have to write code to extend its functionality if you want to do anything but launch an ordinary report (without parameters).

The LocalReport Properties

Let's walk through the LocalReport class to explore the properties, methods, and events you'll likely encounter as you create reports.

  • DataSources: Gets a collection of data sources used by the report. This collection stores ReportDataSource instances that names the Data Source and DataTable used to populate the report. A single LocalReport instance can source data from (virtually) any number of Data Sources. I illustrate how to manage the DataSources collection later in this chapter.

  • DisplayName: Gets or sets the display name of the report. This is rarely set or referenced in the UI.

  • EnableExternalImages: Indicates whether the report can be rendered if it has external images.

  • EnableHyperlinks: Indicates whether the report can be rendered if it contains hyperlink actions.

  • IsDrillthroughReport: Indicates whether the report is a drillthrough report. This read-only property indicates whether the report being launched is a "drill-through" report. A report of this kind is simply a report that's been invoked by referencing it in an action dialog. I illustrate how to set up and manage drill-through reports later in this chapter.

  • OriginalParametersToDrillthrough: Returns the parameters passed from a parent report to a drillthrough report. If the drill-through report needs to adapt its behavior (or query processing) based on parameters passed to the host report, you need to set this property to True.

  • ReportEmbeddedResource: Gets or sets the name of the report embedded resource. This property points to the RDLC file.

  • ReportPath: Gets or sets the local file system path of the local report.

The LocalReport Methods

The LocalReport methods play a pivotal role when it comes to managing report parameters and report behavior.

  • AddTrustedCodeModuleInCurrentAppDomain: Adds the supplied assembly to the list of assemblies that are trusted to execute in the current AppDomain.

  • ExecuteReportInCurrentAppDomain: Causes processing extensions and expressions in the report to be executed in the current AppDomain.

  • ExecuteReportInSandboxAppDomain: Causes processing extensions and expressions to be executed in an application domain with limited permissions.

  • GetDataSourceNames: Returns a string list of the names of all data sources used within the local report.

  • GetDefaultPageSettings: Gets the default page settings specified in the local report.

  • GetDocumentMap: Returns the representation of the document map for the local report.

  • GetParameters: Returns report parameter properties for the report. This method returns a ReportParameterInfoCollection that contains ReportParameterInfo objects. These objects define the report parameters in great detail and are used to pass parameters from the UI to the report processor.

  • GetTotalPages: Returns the total number of logical pages in the report.

  • ListRenderingExtensions: Returns all available rendering extensions for the local report.

  • LoadReportDefinition: Loads a report definition from the local file system. This method can be used to load a RDLC file stored in the local file system.

  • LoadSubreportDefinition: Loads a subreport definition.

  • Refresh: Causes the local report to be rendered with new data.

  • Render: Processes the report and renders it in the specified formatunfortunately, only EXCEL rendering is supported in the RTM version.

  • SetParameters: Sets report parameter properties for the local report. Later in this chapter, I discuss how to create an array of ReportParameter objects and use the SetParameters method to apply it to the report.

The LocalReport Events

There is only one LocalReport eventthe SubreportProcessing event. It's fired when a SubReport item is invoked by the ReportViewer. This event is responsible for supplying data to the subreport. I show how to create subreports later in this chapter and include a long section on how to manage the LocalReport instance to launch custom reports, manage drill-through and hyperlinked reports, as well as manage subreports.

The ServerReport Class

The ReportViewer control can also act as a launching pad for Reporting Services (2005 or later) reports. In this case, the ReportViewer sets the ProcessingMode to Server and exposes a ServerReport class. You'll need to code some extra steps if you intend to set report parameters before invoking the report, so I'll illustrate how to do so next. Remember that, by default, a server report exposes its own parameter prompts. You can choose to override these parameters in code or let the report UI handle them, but that raises a number of complex and undocumented issues.


Managing Server Reports

I'm going to take a bit of a side trip at this point in the tour to show you how to use the ReportViewer to execute SQL Server Reporting Services 2005 reports. This is an important feature that might be overlooked by some developers or architects. In the simplest implementation, invoking an existing server report is merely a matter of knowing what to fill in when prompted by the Reporting Services Task dialog from the ReportViewer control. Let's walk through the basic steps.

1.
Starta new Windows Forms project and drop a ReportViewer control on the form.

2.
Click the ReportViewer Tasks dialog button and select "<Server Report>" from the Choose Report dropdown.

3.
In the Report Server "Url" TextBox, enter the name of the target SQL Server 2005 Reporting Services server that's hosting the deployed RDL report. No, by default, you might not be able to get "localhost" to work as the machine nameeven if the Reporting Services 2005 server is hosted on the local system. This means you'll have to fill in the actual machine name hosting Reporting Services. If your server is configured for encrypted traffic, you'll want to use https://; otherwise, http:// can be used as the protocol. Be sure to end your reportserver name with the instance name. By default, the instance name is separated from the reportserver service name with a "$". In this case, I use https://betav9/reportserver$ss2k5 to address the virtual folder named "ss2k5" hosted on the targeted system's IISin this case, Betav9. Peter is totally opposed to naming these virtual folders with the "$" character to indicate that this folder somehow corresponds with a SQL Server instance name. These can be set as such (with the "$") only during initial setup. This means that if something goes wrong and you have to re-create your IIS folders, you're pooched. I recommend that you get with your IIS administrator and DBA and come up with other, more appropriate IIS virtual folder names.

4.
In the Report Path TextBox, enter the name of the report to render. Note that this name begins with a forward slash (/) and includes the full path to the deployed report. Figure 14.25 shows how the ReportViewer tasks dialog is completed for a server report.

Figure 14.25. Accessing a server report on Reporting Services 2005.

[View full size image]

Rendering the Server Report

When your application that references a server-based report is executed, the ReportViewer control merely addresses the report as it would be referenced from Internet Explorer (IE). It's rendered within the ReportViewer control along with all of the parameter selectors and other options afforded an IE execution. Actually, you don't have to render the report for the ReportViewer control to access the server-side report. When you reference the ServerReport object, the ReportViewer control must fetch the RDL report definition from the server so that it can expose the underlying properties. Yes, this takes a round-trip to the server to fetch, and it does so by executing SOAP protocol commands over the network.

Figure 14.26 is shown by the ReportViewer when executing the server report previously specified.

Figure 14.26. Using the ReportViewer to render a server-hosted RDL Reporting Services 2005 report.

[View full size image]

Note that when rendering any Reporting Services report, all of the output re-rendering options are available (as the drop-down list shown in Figure 14.26 illustrates). In addition, your application did not have to do anything to build pick lists or manage parameters in any way. Since the data queries execute entirely on the remote server, the ReportViewer control only needs to be invoked with the ReportViewer RefreshReport methodyou won't have to worry about building TableAdapter or DataTable objects on the client.

Managing Server Report Parameters

Okay, so suppose you want to handle some of this work yourself in your application. For example, you might want to capture one or more of the parameters yourself instead of giving the user the option of setting them on their own. Before you launch off and begin implementation of this, remember that the Report Manager application supported by Reporting Services can also manage report parameters, as shown in Figure 14.27. Yes, this is discussed in great detail in our Reporting Services book.

Figure 14.27. Managing report parameters using the Reporting Services Report Manager.

[View full size image]

While this is cool, you still think that there are a few more things you want to do with the report parameters and perhaps other aspects of the report rendering. Well, the ServerReport instance is your pathway to the select report parameters and other options. Let's see how you can manage the ServerReport instance configured by the ReportViewer control.

Exploring ServerReport Parameters

To explore the parameters used to drive this server report, I coded a routine to dump the contents of the ServerReport parameters collection as returned by the GetParameters method. It turns out that a GetParameters method is also exposed on the LocalReport. The code for the routine to explore the parameters is shown in Figure 14.28. Note that the GetParameters method returns a collectionthe ReportParameterInfoCollection, which contains ReportParameterInfo objects. Each of these objects describes the individual parameters. For example, you'll find the parameter name, the valid values, and the currently set value(s). Yes, a parameter can have more than one value, as Reporting Services 2005 and the ReportViewer support multi-valued parameters. I'll discuss this aspect of parameters later in this chapter. Sure, there are a lot more properties to explore here, and I'll leave that adventure to you and your spare time.

Figure 14.28. Dumping the contents of the server report parameter definitions.

[View full size image]

The server report I showed you earlier contains the following parameter settings (as shown in the ListBox on the left of the form in Figure 14.29). This dump tells us a lot about the parameters required for this report but also gives us a number of hints as to how to manage them in code. Remember that Reporting Services generates the UI elements used to prompt and capture values for the report parameters.

Figure 14.29. Exposing the server report parameter configurations.

[View full size image]

If you want to manage these parameters yourself, you might want to disable the report parameters UI section. This is easily done by setting the ReportViewer PromptAreaCollapsed property to True. However, if you don't disable the "Show Prompt Area" button, your user can simply click on the button to re-expose the parameter prompt area. This means you'll need to set the ShowPromptAreaButton property to False. No, these properties don't make much sense when you're working with a local (client-side) report unless you want to use them to show or hide your own home-grown prompt areas.

Next, if you intend to manage parameters yourself, you need to add UI elements or code to set the report parameters. Given the report I'm using in the example, this could mean that your code would need to implement (at most) 12 UI elements (6 labels and 6 TextBox/ComboBox controls) to prompt for and capture the parameter values used to execute the report. This can be very tedious and expensive if you consider that you might have hundreds of reports to manage with the ReportViewer control. To deal with this issue, you might want to generate the UI elements in much the same was as Reporting Services doesgenerate the UI elements automatically from the ReportParameterInfoCollection. I've done just that in the example written for this section (prjServerReports). Figure 14.30 illustrates a simple implementation of this approach. In this case, I walked the ReportParameterInfoCollection and picked off the parameter name and value(s). These were placed into Label and TextBox controls generated on-the-fly.

Figure 14.30. Generating Parameter prompts and TextBox controls to capture new values.

[View full size image]

Sure, you might want to selectively choose which parameters to display, but this code should get you started. Part of the code used to generate these UI elements is shown in Figure 14.31. It's generic enough to permit you to generate UI elements from any Reporting Services report loaded into the ReportViewer. See the example on the DVD for the rest of this routine.

Figure 14.31. Building the parameter TextBox UI elements on-the-fly.

[View full size image]

Yes, you could (and probably should) enhance this code to validate the values entered into the ComboBox controls to conform to the permissible values stored in the ReportParameterInfo objects. By using ComboBox controls, I have taken the first step toward that goalonly permissible values are listed in the control.

Resetting Parameter Values

Once new values are set, you'll need to reset the ReportParameterInfoCollection with the values. No, you don't need to change all of the defined parameters, but you will need to set them all. That is, even though you aren't changing a specific parameter, you'll still need code to "touch" each parameter. That's because the SetParameters method accepts a fully populated array of parametersall of them, not just the ones you've changed.

The routine shown in Figure 14.32 walks the array of controls I generated to display the parameters to the user and rebuilds a collection of ReportParameter objects that contain the Name and Value of each parameter.

Figure 14.32. Building an array of ReportParameter objects to pass to SetParameters.

[View full size image]

Interesting ServerReport Parameters

As I wrote this chapter, I came across any number of ServerReport properties (some of which are underdocumented) that you might find interesting or perhaps essential as you manage parameters or other aspects of the ServerReport. These include:

  • DisplayName: Sets the "display name" of the reportwhatever that is. This is pretty much undocumented.

  • HistoryID: This property is used to specify that you want to render a specific report historical snapshot. Each of these reports is identified with a HistoryID.

  • IsDrillthroughReport: Indicates that this report is a drill-through report. See the discussion of drill-through reports later in this chapter.

  • ReportServerUrl: This property points to the Reporting Services server and SQL Server instance.

  • ReportPath: This property reflects the name of the report you entered when filling out the ReportViewer Tasks dialog.

  • ReportServerCredentials: Used to determine the credentials to use to access the Reporting Services. This property is used to manage SQL Server rights access to the report server.

  • Timeout: This determines how long the ReportViewer waits to render your report. The default is 600,000ms (10,000 minutes, 6.94 days, or the time it takes you to figure your taxes if you're filing a 1040A).

Interesting ServerReport Methods and Functions

I also came across a number of (also underdocumented) ServerReport methods. While it's beyond the scope of this book to get into these in any detail, I hope to supplement the book with a few EBook chapters that dig into the inner cavities of the ServerReport class. I also found several methods and functions that were simply undocumented. However, I expect that you'll find these documented methods as intriguing as I did. Note that all of these points are discussed in depth in our Reporting Services book.

  • LoadReportDefinition: This method reads a data stream and installs it as a report on a specified Reporting Services server.

  • Refresh: Basically repeats the RefreshReport method of the ReportViewer controlcauses the report to be rendered with new data.

  • Render/RenderStream/ListRenderingExtensions: Permits you to send the report to a targeted rendering extension.

  • SetDataSourceCredentials: When the Reporting Services Data Source is configured to require credentials, this method is used to pass those credentials.

  • SetParameters: This reloads the report parameters as demonstrated and discussed earlier in this chapter.

As for the ServerReport functions:

  • GetDataSources: Returns a collection, ReportDataSourceInfoCollection, that describes the Data Sources used to populate the report. Ah, in my tests, this method didn't return anything.

  • GetDefaultPageSettings: The page settings are returned in a ReportPageSettings structure that exposes the Margins and PaperSize defined for the report.

  • GetParameters: Again, I just discussed this function earlier in this chapter. It's used to fetch the list of parameters from the RDL report definition.

  • GetServerVersion: Yup, this is interesting, but since the ReportViewer works only against SQL Server 2005, it won't be of much interest until another version ships. The current server version reports 2005.090.1399.00. I guess that's specific enough.

  • GetTotalPages: This returns (you guessed it) the total number of logical pages in the report.

As you explore the rest of this chapter, consider that this approach can also be used when building generic parameter-management routines with local reports.


Managing Parameters

At this point, I expect you to understand that your reports are going to need to manage, limit, sort, filter, and focus the report rowset using parameters. Because of the nature of today's applications, users expect applications to be interactivethey rarely settle for static reports that don't reflect current or user-selectable sets of information. In order to adapt reports so they meet user requirements, you're going to need to use one or more parameters. Basically, ReportViewer parameters are organized into two basic types:

  • Query parameters: These parameters are passed to the data source query engine to alter the TOP, WHERE, or ORDER BY or other SQL clauses to change which or how many rows are returned from the database query. Unlike RDL-based reports, the query parameters are not stored in the RDLC report description. However, when the ReportViewer is used to host remote reports, the RDLC maps a report parameter to each query parameter. In this case, your code needs to populate these parameters before launching the report.

  • Report parameters: These parameters are used to alter the report properties and behavior at runtime when working with RDLC reports. For example, I could have set a "ShowColor" property to let the user choose whether or not to show the product color background in color. These parameters can also be used to filter, sort, and refine the rowset data using post-fetch techniques, as I discussed earlier. When working with remote RDL reports, the report parameters serve to expose the underlying query parameters, as well as those parameters used to feed report expressions.

One of the challenges you'll face when working with the ReportViewer paradigm is that all parameter-management tasks fall on your shoulders. Unlike the Reporting Services BIDS development process, RDLC reports do not manage the report or query parameters for you in the context of the report. Remember Figure 14.3? The "Low Price" parameter TextBox was generated by the RDL report interfacenot by a hand-coded UI. The BIDS Report Designer is capable of managing a wealth of parameter optionsnone of which requires additional code. Even if your RDLC file is built from an imported parameter-driven RDL report definition, the report parameters section is ignored unless you build a custom Parameters structure in code to populate it. This means you're in for considerable extra work, including (but not limited to) capturing the parameters, managing pick lists, and passing the parameters into structures recognized by the RDLC compiler. Let's get startedI'm losing daylight.

IMHO

If wishes were horses, I would need to clean the streets more often.

One of the tools that might make sense is a routine that generates the UI controls needed to manage parameters for any given RDL report. Perhaps, one of us will write that if we get bored.

Capturing Parameters

Parameter values can come from anywhere (except Cleveland and parts of south Texas). They can be provided by the user punching keys or doing cut/copy/paste mouse operations, from code-generated values, from data retrieved from a pick list, from external devices like a GPS coordinate generator, or from your cat walking on the keyboard. Ideally, your report (and your application) avoids scenarios where the user is required to enter anything into TextBox controlsespecially if these are values that are concatenated into your SQL. That's because SQL injection attacks are a very common problem, and report parameters are an ideal place to enter nasty SQL.

Let's walk through the typical parameter-management scenarios and how to implement them.

Prompting for a Parameter Value

The easiest (and possibly most dangerous) way to capture a parameter values is to place a TextBox in your UI and ask the user to enter the value. Figure 14.33 shows a ToolStrip containing a prompted TextBox used to supply a value to a parameter that limits the scope of the rows returned for the report.

Figure 14.33. Using a prompted TextBox to capture a parameter.

Here, the user can enter (almost) anythingas long as it's a number between 0 and 10,000. Behind the scenes, I've done a number of things to make sure this value is "reasonable" (falls with the specified values) and safe. These include:

  • Setting a reasonable default value. Ideally, the last value entered should be retained in the user settings so when the user returns to this application, the value last entered is the default.

  • Limiting the type and scope of the object value captured by the TextBox. One way is to simply test the value as shown in Figure 14.34. This routine validates the Text property to ensure that it's numeric and falls within a reasonable range.

    Figure 14.34. Validating the user-input string.

    [View full size image]

  • Make sure that the value is applied to an ADO.NET Command object's Parameter objectnot simply concatenated to the SQL. This simple step goes a long way toward preventing SQL injection attacks.

Prompting with a Pick List

In cases where the parameter can be only one or more specific items, a pick-list of some kind is the best way to capture the parameter value. Generally, you'll want to populate the pick list early in the application if the content of the list does not change that frequently. If the list changes more frequently, of course, the pick list of valid options will have to be populated more frequentlyperhaps each time it's referenced.

One of the new features of Reporting Services 2005 is the ability to expose one or more "multi-choice" parameter lists. In this case, Reporting Services knows how to present and populate a list of choices that can be populated via a static list or a query. It's this functionality that I'm implementing here with my own code and controls.

If you use the query approach to populate a pick list, you'll have to make an extra round-trip to the data source to fetch the values. Alternatives to this query can be a locally persisted data store of some kind that maintains a suitable list, but this means building your own handlers, files, and many other issues. I find it's not at all onerous to simply fill the pick lists from a common command that includes SELECT statements for all of the pick-list rowsets. Of course, if you're using JET or some other lightweight database that does not support multiple SELECT commands, you'll need to find another alternative, like paper tape.

For this example, I built two types of pick lists:

  • A query-populated ComboBox (incorporated into a ToolStrip control). This approach lets the user pick one item from a drop-down list. This list is populated from a TableAdapter that returns all known product colors.

  • A custom dialog that contains a CheckedListBox control populated with data returned by the same TableAdapter used in the ComboBox population. I could not incorporate the CheckedListBox into the ToolStrip, as it is not supported.

No, I did not have to use a TableAdapter to fetch the rowset used to populate these pick listsfrankly, it was almost as much trouble as using my own Connection, Command, and DataTable instancesbut I'm trying to leverage this new technology and it did take fewer lines of code.

The ToolStrip control with its two pick lists is shown in Figure 14.35. I find that the ToolStrip is a pretty cool control, as it provides a clear and intuitive way to generate the needed UI elements. The problem with this approach is that it's harder to bind to the ToolStrip-hosted controls, so we ended up having to populate the lists ourselves.

Figure 14.35. Using a ToolStrip to host pick lists.

The populated CheckedListBox is shown in Figure 14.36. Note how both pick lists include an "<All>" item. This permits the user to indicate an ambiguous item choice. In the query that returns the filtered rowset, I include logic in the WHERE clause to ignore the pick list items and simply accept any color value when "<All>" is passed as a parameter.

Figure 14.36. The populated CheckedListBox pick list.

In the case of the CheckedListBox or with any ListBox-based pick list, you don't have to worry about the user entering invalid entries. However, with a ComboBox, you do. This means you still need a validation routine to make sure the entry chosen (or entered) is still valid.

Creating a Pick-List Query

By now, you should be very familiar with the techniques used to return a rowset. That's all you need to populate you pick-list control. In the example, I create a TableAdapter to return a list of the valid colors. This query is executed by creating an instance of the TableAdapter and a strongly typed DataTable, and executing the TableAdapter instance Fill method to populate the strongly typed DataTable. I extract the list of colors from the Rows collection and move it to an array of objects acceptable to the ComboBox (hosted in the ToolStrip) and the CheckedListBox (hosted in its own dialog).

Figure 14.37 shows the code to generate the object array used to populate the pick lists. Note that I add "<All>" to the list (line 13) so that all color lists support the option to select all items.

Figure 14.37. Fetching the list of known colors.

[View full size image]

Passing the Picked Items to a Query Parameter

Passing a single value to a T-SQL query is very easy. One simply creates a parameterized CommandText and a suitable Parameter instance for each marked parameter, and sets the Value property of each Parameter to the item chosen from the pick list. That is, unless the user has selected a list of items (chosen more than one item from the list of possible choices). In this case, one needs to either generate the T-SQL WHERE clause on-the-fly (as it appears that Reporting Services is doing) and concatenate it to the rest of the SELECT query or leverage the WHERE clause IN operator. As I've discussed before, it's not possible (due to SQL syntax limitations) to pass a parameter to an IN operator. To deal with this shortcoming, I pass a table populated with the items to the IN operator.

The trick is to populate this (temporary) table from the delimited list of selected items chosen by the userthat's what my CLR function DelimitedStringToTable does. I deployed this commonly used CLR function to the Common database so that it could be accessed by any database that needs it but still could be centrally managed. Of course, this assumes that your DBA will let you near the "enable CLR" switch. Most DBAs have this held in the off position with a dozen layers of packing tape.

Tip

I use a "common" database to host functions that can be used by more than one database.

To return the "filtered" set of row for the report, I created another method (FillByColor) on the TableAdapter used in the previous (parameterless) example. The parameter-driven query is shown in Figure 14.38. Note that it uses the CLR function drawn from the Common database to convert the delimited list to a temporary SQL table used by the IN clause.

Figure 14.38. The Parameter-based FillByColor used to return rows that match one or more colors.

[View full size image]

Sure, there are lots of other ways to limit or focus your report rowset query. These include (but are not limited to):

  • Using the WHERE clause to focus the rows using specific criteria either passed by parameter-driven expressions or generated by other means.

  • Using the TOP clause to cap the number of rows returned by the query based on a value or simply the top N percent of the available rowsets. This is a great way to rank rowset valuesreturning the top 10 salespeople or the top 5% of the stores based on volume.

  • Using the WHERE clause IN operator, as illustrated earlier, to include a list of items that further refine the inbound rowset.

Running the Report Query

Yes, unlike Reporting Services, your application code is responsible for executing the report queriesthose that populate the pick lists and, ultimately, those that return the rowset used to populate the report RDLC fields. You decide when these queries need to run, but I suggest that they not be executed in Form_Load, as this delays the painting of your form and means the user stares at a blank screen until the query is executed and the report processed. Generally, your application should collect any focusing parameters from the user before launching off on a report query that might take several minutes to the rest of the afternoon to execute.

Note that all of the techniques I describe here execute code that executes the query and returns a DataTablenot just a DataReader. This operation completes rowset population of the DataTable, which is ultimately passed to the ReportViewer control while it's processing the report definition (RDLC) file. This DataTable can be created via a TableAdapter (as I've done so far), by building a DataTable from a DataReader, or by using a DataAdapter Fill to populate a DataSet with one or more DataTable objects. Later in this chapter, I'll discuss some of the advanced techniques that permit you to bind your own DataTable object to the ReportViewer. In any case, the rowset returned from the report query must contain the Fields referenced in the RDLC report definition. Sure, the DataTable can contain extra columnsbut it must return columns or expressions that match the names and types of the Fields referenced in the RDLC report definition.

Report Parameters

If you recall, a bit earlier in this chapter I mentioned a second type of parameterthose used by the RDLC to filter, sort, and alter the behavior or how information is shown in the report. These are called "report" parameters. Remember, report parameters map only to expressions in an RDL report used by the ReportViewer in local mode. Yes, report parameters also map to query parameters when using the ReportViewer in remote modeas when the ReportViewer is hosting an RDL Reporting Services report. The techniques used here apply to both ReportViewer modes.

Basically, report parameters are exposed via the ReportParameterInfoCollection instance exposed by the LocalReport instance. This structure contains an array of ReportParameterInfo objects that define (in infinite detail) each parameter defined for the report. With its properties, you can construct a sophisticated prompting dialog to capture the values to be passed to the reports that consume the parameters. Yes, parameters can be "multi-value" and have more than one value. For example, if you're building a dialog to capture valid state codes, but your report logic permits more than one selection, you can designate the parameter to accept several values.

Creating report parameters is accomplished via the Report Parameters dialog exposed by the Visual Studio IDE when focused on an RDLC report definition. This dialog (as shown in Figure 14.39) permits you to name the parameter and set the datatype, the prompt (if needed), as well as default values. Unlike with Reporting Services, you can't automatically populate the parameter permissible values at runtimeat least, not automatically. And no, the ReportViewer control won't prompt for parametersit's up to you to add logic to your host Windows Form or ASP page to capture the values.

Figure 14.39. Constructing a report parameter.

[View full size image]

Let's step through the scenario I just demonstrated to set the lowest acceptable profit and the desired list of colors, but in this example, I'll use report parameters to do the same job. This means I'll accept far more rows from the query and filter "out" those rows based on report parameter settings. The advantage to this approach is that once the report is rendered for the first time, no subsequent trips to the data source are needed to fetch subsets or to reorder data. Sure, you still might need to capture the values of the report parameters using UI elementsthat part has not changed. The only part that will change is how these values are applied to modify the behavior of the report.

1.
Start from the Products By Profit example report I've been working with (prjLocalReport). I copied one of the existing reports to build ProductByProfitRptParm.rdlc.

2.
Select the report in the Report Designer and open the Report | Report Parameters menu item. This exposes the dialog shown in Figure 14.39.

3.
Set up a new report parameter as shown. No, you don't need to define the Prompt property unless you plan to use the parameter definition to feed a UI element to capture the parameter value. It's okay to specify a default value, as this makes debugging simpleryou won't have to define a "real" value until you're ready.

4.
Select the Table report item in the Report Designer and choose Properties and then the Filters tab. This opens the dialog shown in Figure 14.40. Note that I already have several filters in placethe first hides rows that have no color defined. The second filter makes sure that the cost and list price are not NULL. The new filter stipulates that all rows where the profit is less than the minimum acceptable value are hidden. This value is extracted from the MinimumProfit report parameter at runtime (when the report is rendered). Note the somewhat peculiar syntax. This is often (usually) necessary because the RDL/RDLC compiler is a bit anal about datatypes. It requires that both the Expression type and the Value type be the same. This means it's best to simply code an expression that resolves to True or False and test for that Boolean state in the Operator/Value pair.

Figure 14.40. Setting the Filter property to limit visible rows based on the report parameter.

[View full size image]

5.
Next, I need to set up a dialog to capture the minimum profit report parameter value at runtime and feed it into the LocalReport parameters. I accomplish that with a label and TextBox control on the Windows Form, and I'm ready to go...well, almost.

6.
In order for the ReportViewer to use the parameter, I have to construct a ReportParameter instance and populate its value with the UI TextBox value. The code shown in Figure 14.41 does this.

Figure 14.41. Populate the ReportParameter instance and set the LocalReport parameters collection.

[View full size image]

The SetParameters method cannot be called until the RDLC file is set. This method also validates the parametersthey must exist and conform to basic type validation. This means you'll have to be sure to pre-define the parameters (as described earlier) before trying to apply a value. Basically, this process maps the existing RDLC-defined report parameters with a matching array of ReportParameter instances.

Report parameters can be used in expressions all through the RDLC report definition. This means you can use them to manage the look and feel of the report, filter data in (or out), customize the text boxes used in headings, or anywhere else and expression is exposed.


Advanced Reporting Techniques

This section discusses a few of the more complex aspects of report creation and management, including how to set up click-through reports and how to include subreports in your reports. These topics are included, as they're programmed entirely differently than the techniques discussed in our Reporting Services book.

Managing Click-Through Reports

One approach that you're going to want to try is creating a "click-through" report. Imagine, if you will, a report that displays limited information about a productbut enough for most uses. However, to fetch additional information, the user is prompted to click on a specified field (as shown in Figure 14.42). This launches a "click-through" report (as shown in Figure 14.43). The magic here is that the ReportViewer is automatically reprogrammed to point to the targeted drill-through report and pass zero to many parameters to the report. That is, a new ReportViewer or Windows Form is not createdthe existing form and control are reused. This means that your user will have to use the "back" button to return to the host report. Thankfully, this transition is handled automaticallyas long as you use the "back" button. Sure, if you want to pass parameters to the drill-through report, you're going to need to write some code to fetch the parameters and run a custom query. I'll show how that's done later in this section.

Figure 14.42. A report hosting a drill-through report.

[View full size image]

Figure 14.43. The drill-through report.

[View full size image]

Note that this "drill-through" approach is different from a SubReport (that I discuss later in this chapter). In contrast, a SubReport is embedded in a Table report item cell or co-located on the report surface. In either case, you'll need to provide a fresh DataTable and Data Source to connect the report items with the data that populates them.

When accessing a report programmed for drill-through operations, once you click the pre-programmed Table report item cell, the ReportViewer launches the specified (possibly parameter-driven) report to show more detailed information about the selected cell, as shown in Figure 14.43. Behind the scenes, you might have to trap the ReportViewer Drillthrough event, where you execute the (parameter) query used to populate the drill-through report. I'll show you how next.

Okay, this seems cool, but getting to this point is a challenge, given the sorry state of the documentation. Let's walk through the process of getting to this statestep by step.

1.
Start a new project. I created prjClickThroughReport to illustrate the final product.

2.
I leveraged one of the previous reports that returns product information and created a new StockReport to launch as a drill-through report. Add these to your project along with appropriate Data Sources. I used two strongly typed TableAdapter classes to return the initial rowset as well as the parameter-driven drill-through report data. The StockReport query is customized to accept a ProductIDWanted parameter that focuses the returned rowset on the specific rows in the Inventory table for that product.

3.
Since this report is populated from a parameter-driven query, I need to specify a report parameter to pass along to the custom Fill method. Open the drill-through StockReport and choose "Report Parameters". Fill in the dialog with the Parameter name and default value, as shown in Figure 14.44. No, the prompt is not required in this case, as you won't be prompting the user for the valuebut you could if that seems more appropriate. Note that you set a default value, but this also is not used unless creating a user interface to capture the value from your human operator. By default, the parameter value is filled in automatically from the cell I'm about to program on the host Table report item value.

Figure 14.44. Defining the drill-through StockReport report parameter.

[View full size image]

4.
Next, open the host report, select the detail row "Product Number" column (as shown in Figure 14.42), right-click, and choose "Report Properties". I'm about to set up the linkage between the host report and the targeted drill-through report, and define the parameter to pass to the drill-through report.

5.
Choose the "Navigation" tab and select "Jump to report:". Choose the targeted report from the drop-down list, as shown in Figure 14.45. This instructs the ReportViewer to reconfigure the LocalReport class to address the specified report when the cell is clicked at runtime.

Figure 14.45. Programming a Report Table item cell to redirect to a drill-through report.

6.
Since the drill-through report requires an input parameter (to focus the displayed results), click on the "Parameters..." dialog. This opens another dialog where you can specify how the report parameter required by the drill-through report is generated.

7.
Open the drill-through report (StockReport) in the Report Designer, right-click, and choose "Report Properties". This opens a dialog, as shown in Figure 14.46, that permits you to map each report parameter (in this case, ProductIDWanted) defined in the targeted drill-through report with a value. This value can be a constant or an expression, or can be drawn from an existing Fields value (as shown). In this case, I bound the report parameter to the current value in the selected row's Table item cell containing the ProductID. At runtime, when a Product ID cell is clicked, this value is passed to the drill-through reportas long as you write to code to do so. Click OK to save these settings and OK to close the TextBox properties dialog.

Figure 14.46. Binding the drill-through report property to the parameter value.

No, I'm not really donenot nearly. While the automatic linkages would work if I were not executing a special parameter query, I need to step into the process of launching the drill-through report and manage the parameter query used to populate the StockReport.

8.
Get into the code editor and set up the ReportViewer Drillthrough event handler. In this example (as shown in Figure 14.47), I first set up a variable to handle the report parameters (the ReportParameterInfoCollection). This structure contains a wealth of information about the report parameters that can be very useful when building UI to capture the parameters from your report user.

Figure 14.47. Managing the Drillthrough event.

[View full size image]

Next, I pick off the LocalReport instance passed to the event handler in the DrillThroughEventArgs (line 19). This addresses the targeted drill-through report. I test to see if this drill-through report is the one I expect, and if this is the first time the event is called, I instantiate a new TableAdapter and DataTable to fetch and hold the rowset.

9.
The next block of code (shown in Figure 14.48) illustrates how to pick off a named parameter from the ReportParameterInfoCollection. I use the LocalReport GetParameters method to fetch the parameters (line 28). Note that this method also validates the parameter values, so if they don't comply with the datatype specified in the report parameter definition, the method throws an exception.

Figure 14.48. Retrieving the report parameter and executing the custom Fill method.

[View full size image]

If for some reason no value is passed, I use a default parameter value of 0; otherwise, I set the inbound parameter value to the named parameter value. Note that the ReportViewer knows how to manage value arraysthat is, a parameter value can be the result of the user selecting several items. The UI setup does not support this, so I simply pick off the first (and only) value (line 33).

10.
The next block of code is responsible for executing the custom Fill method (FillByProductID) and populating the DataTable passed to the ReportDataSource. Note that the ReportDataSource name must match the DataSet Name element in the RDLC definition. Ordinarily, this is set when you drag columns from the Visual Studio Data Source to the report elements, but if you import the RDL from another source, it might be different.

Remember that when you use this approach, the user has to press the "back" button to return to the host reportunless you provide your own navigation buttons. Any Table item cell or TextBox item can be programmed as a "button" or hot-spot by setting the navigation properties.

Sure, if you want to support additional cells that drill through to other drill-through reports, you'll have to create logic in the Drillthrough event handler that checks to see which drill-through report is being requested. In the final version of the prjClickThroughReport application, I implemented this functionality to illustrate this, as shown in Figure 14.50. Once this is configured, the targeted drill-through report can act as a host and launch additional drill-through report(s). Remember, the ReportViewer control is maintaining a "stack" of reports so that your user can navigate back to the original report.

Figure 14.49. Executing the custom Fill method and setting the Data Source.

[View full size image]

Figure 14.50. Branching based on the drill-through report name requested.

[View full size image]

Using the SubReport Report Item

Another approach you can implement to show additional detail leverages the SubReport item. This technique is a bit easier to implement when compared to the drill-through reports I just discussed, but it includes a number of other nuances that I need to discuss.

The example application is similar to those I've already shown you, but in this case, I create a subreport that displays the available photographs of selected products. The subreport can be positioned in a number of places, including on the report surface or in a Table item cell. I chose the latter for this example, as it best fits with the report's interactive "finder-detail" design. That is, the host report simply lists products (that have photos) but does not show the pictures for each row. Even though the pictures are not initially shown for each item (as shown in Figure 14.51), the ReportViewer invoked the SubReport for each row returned in the Products query. This is pretty expensiveone round-trip query for every row in the Products table.

Figure 14.51. The host report with the SubReport item hidden.

[View full size image]

In this case, I set the Visible property of the SubReport using a TextBox cell in the Table item. When the user clicks on the "+" sign, the ReportViewer toggles the SubReport Visible property. This is programmed via the SubReport item properties page, as shown in Figure 14.52. Sure, you can write an expression to hide the SubReport (or most items).

Figure 14.52. Programming the visibility of the SubReport item.

Once the user clicks the "toggle" item in the Table item cell, the SubReport item is made visible (as shown in Figure 14.53).

Figure 14.53. Toggling the Visible property of the SubReport.

[View full size image]

No, this is probably not the best way to implement a subreport. A more efficient approach would be to construct the report so that the SubReport was not rendered along with each row fetched. In this case, the SubReport would be placed outside of the Table cells. The problem with this approach is that, unlike the drill-through report, you can't launch it from a cell in the Table report item. For the example (shown in Figure 14.54), I created a report that generates a small chart that reflects the overall sales of this product line. In this case, the code to generate its rowset is called only oncewhen the report is first rendered.

Figure 14.54. Rendering a SubReport item outside of the Table.

[View full size image]

As you can see, I used the Chart report item in the last SubReport example. I devote an entire chapter to the Chart report item in our Reporting Services book. There, you will find details on how to configure the Chart report item to generate a wide variety of charts and graphs.

Note that Reporting Services 2005 now supports "control extensibility"that is, third-party vendors can (and have) adapted their custom report items to Reporting Services 2005. For example, the Dundas Chart report item has been so adapted. Unfortunately, these third-party report items do not work with the ReportViewer control. That said, it really isn't that much of a limitation. Because you have control over data extraction, you can use custom-built controls to return binary image streams and consume them within the image control. Conceptually, this is not much different from how the custom report items work.

What's in the RDLC?

I've often had to dig into the report definition file to debug one problem or another or resurrect a dead or crippled report. The Report Definition Language (RDL) variation generated by Visual Studio for use with the ReportViewer control is not that different. It has a different extension (RDLC) but shares a great deal of functionality with the BI-tools generated RDL files. The "C" stands for "client" as these reports are designed for execution on the clientnot the server by Reporting Services. Remember that while the RDL and RDLC report file contain information about the SELECT, the Data Source, and even the ConnectionString, the client-side ReportViewer control does not use all of these elements in the same way.

The <DataSets> Block

The core of the RDLC file is the DataSets block as shown in Figure 14.55 Note the presence of entire query (the CommandText attribute), references to TA Fill and GetData methods, SELECT, and DataSourceName and ConnectionString in the <DataSets> block. Many of these sections in the RDLC are required for it to compile (a process that's repeated each time against each RDLC file once you start the application). The <CommandText> element (this contains the SELECT statement) and the <rd:DataSetInfo> sections do not need to be filled in, as they are not run by the ReportViewer. These methods are up to your code to runnot the ReportViewer.

Figure 14.55. The RDLC report definition dumped to XML.

[View full size image]

Note that the DataSet Name element must be present and must point to the DataTable (or equivalent)it's referenced in the LocalReport instance. Fields are mapped to report Fields but are otherwise not referenced. The values in this block are set when columns are dragged from the Data Source window to the report designer report items. These settings would be used if the report was to be rendered by Reporting Services, but since they're generated from a strongly typed TableAdapter, they can't be run on SQL Server Reporting Services as is.

Converting RDL to RDLC

Okay, so you got your boss to sign on to Reporting Services and you've invested a couple of months (or a year) creating reports using the BI tools or Visual Studio 2003's Reporting Services add-in. Now, you want to leverage that work with the ReportViewer control. Of course, the simplest thing to do is launch the server-side report using the ServerReport mode of the ReportViewer control. If you need to execute the report on the client for any reason, you'll need to convert the RDL to RDLC format. Unlike some other XML fiascos (as when you tried to use ADOc Recordsets persisted as XML in ADO.NET), the transition to RDLC is really pretty easy because the schema for both files is identical. While the ReportViewer control ignores the <Query> element, the other elements and attributes are processedalbeit in somewhat different ways.

Basically, to convert an existing RDL report to RDLC, you're going to need the RDL XML file. This can be extracted from SQL Server by using the Reporting Services Report Manager or by bribing the developer that owns the file. Ah, only Reporting Services 2005 RDL files can be imported and converted to RDLC format. If you're working with Reporting Services 2000, you'll need to use the BI tools to open each RDL report and let the tools convert the RDL to the new format. Of course, this is a one-way conversionthere is no tool to convert them back to Reporting Services 2000 format.

Once you have the RDL (2005-format) file, you need to follow these steps:

1. | Rename the .RDL file to .RDLC.
2. | In Visual Studio 2005, open the solution or project that's intended to host the report.
3. | Create the DataSet that returns the data using the same SELECT query defined in the reportit must return the same columns with the same names as the RDL SELECT. You can create the DataSet by any means you chooseusing the wizards or simply by rolling your own ADO.NET 2.0 code. This SELECT is contained in the RDL file in the <DataSets> element. Yes, there may be more than SELECT, so while this might be easy for a simple report, it can be a lot of work for a report that requires a number of queries. However, the RDL report can also contain queries used to populate pick lists, and while you don't need these to convert the report, your ReportViewer program code might need to implement these pick list queries along with the UI elements that are supported automatically in the report.
4. | On the project menu, click Add Existing Item, navigate to the RDLC file and click "Add". Yes, you'll have to change the "Files of type:" entry on the explorer to see the RDLC file you're converting.
5. | In the project, open the Form containing the ReportViewer control that will host the converted RDLC report.
6. | In the ReportViewer Tasks smart tags panel, select Choose Reports and select the .RDLC report you've just imported into the project.
7. | While still in the ReportViewer Tasks smart tags panel, select Choose Data Sources and choose the DataSet you want to use to populate the report.
8. | Save all of the files and test the report.

For more information and details about how to convert RDLC reports to RDL format, see http://msdn2.microsoft.com/en-us/library/ms252109.aspx.


Implementing the Matrix Report

This next section steps a bit deeper into report design and data modeling as I create a fairly sophisticated (in comparison to the reports I've already discussed) report using data from the Biblio database. This report is designed to visually analyze the total financial value of orders (OrderValue) and the number of orders (OrderCount). For this example, I'll use the Matrix report item to break down the data by U.S. state (State) to which those orders were delivered over time (Year, Month, Week, Day).

The Data Considerations for a Report

As with any serious reporting project, it's vitally important to understand how to best configure the rowsets that serve as the fodder for the reporting engine. Ideally, to extract this data from the Biblio database, you would issue an aggregate query. This means in T-SQL, you would use a GROUP BY clause. As you'll discover, the Matrix report item and the other ReportViewer report items used in this example can themselves perform grouping and sorting. However, in most cases, it's best to perform any grouping at the data source. Yes, it's best to let the DBMS engine do what it's designed to dogroup, sort, and return the pre-configured rowset products, especially considering that the database has indexes that can assist in the optimization of grouping queries (or, it had better have).

To keep things simple, let's build up the query without any grouping. For this example, I need to target and process four tables in the Biblio database. These include the Items, Orders, Addresses, and ValidStates tables, as shown in Figure 14.56.

Figure 14.56. The Biblio database tables used in the Matrix example.

[View full size image]

Let's take a closer look at some of the fields that must be built from expressions in the server-side query. Sure, composite column expressions can be done on the client by the ReportViewer, but it often makes sense (as it does in this case) to let SQL Server do the work.

  • OrderValue: This field can be processed from the Items table and will be simply a calculation of the QtyOrdered field multiplied by the Price field:

    Items.Price * Items.QtyOrdered

  • OrderCount: For the moment, let's ignore OrderCount and just return the OrderID. As you'll see later, counting orders is best done with the T-SQL aggregate function COUNT().

  • State: To extract a human-readable State Name, we need to JOIN the Orders table to the Addresses table to the ValidStates table. If you are really astute, you'll see that I have LEFT OUTER JOINS starting from the Orders table. Actually, in this, case, it may have been better for us to do equivalent joins, but that would depend on us having full confidence in the referential integrity (RI) of the database and the cleanliness of the data. For example, if there are any orders shipped overseas, their addresses would not have a valid StateCode value. If I used an equivalent join, I would drop rows that I would ordinarily include in the report. In financial reporting, dropping odd rows can have dire consequences. For example, if your report understates the company's total sales, the CFO might decide to axe your subsidiary because it's not profitable.

  • Year, MonthName, MonthNumber, WeekNumber, DayName, DayNumber: In the report that we are eventually going to build, having the DateOrdered field broken down to Day will be very useful. The T-SQL expressions shown in Figure 14.57 are used to generate highly granular data values from the DateOrdered field.

    Figure 14.57. Using T-SQL to expose the DateOrdered field.

The Initial Select Query

I'm ready to make a first cut at the SELECT query used to drive the report. This code is shown in Figure 14.58. Yes, this T-SQL will eventually end up in a TableAdapter query or (better yet) in a stored procedure that's executed to return a DataTable to be bound to the new report.

Figure 14.58. The initial version of the report query.

[View full size image]

If you execute this query against the Biblio database, you'll find that you get back about 876,787 rows. That's really an awful lot of rows. Retrieving this behemoth rowset takes several minutes to execute query and to push those near million rows across the networkassuming the query is being executed on a remote server (as is usually the case when using the ReportViewer control). In this case, it makes abundant sense to perform as much aggregation on the database server as possible before pushing the rowset out over the networkespecially when you consider that, for this report, I don't need line-item detail.

Grouping the Data

By grouping the data on the server (as shown in Figure 14.59), I reduced the rowset to a mere 76,845 rows. That's still pretty big, but about [image] as large as the ungrouped rowset. Sure, many reports require you to build gargantuan rowsets that your client application must accommodate in memorynot to mention the load placed on your network to transport the rowsets to the client.

Figure 14.59. Using SQL Server to group the data for the Matrix report.

[View full size image]

Grouping on the server does yield performance improvements, but not enough for us to be happy in an on-demand report in a real production system. The ungrouped query took about 2 minutes to execute on one of my machines, and that machine had little else that it was trying to do. Goodness knows how it would have performed if more than one person ran the query at the same timewhich would happen if more than one person asked for the same report at the same time.

In contrast, the grouped query took about 22 seconds to execute. That's quite an improvement but still a strain on the system resources, as I saw the CPU heat up and go to 100% utilization for most of the time. It seems to me that this is still a long time for report users to waitand I still haven't rendered the report. If this report was executed simultaneously by more than one user, the time would have started to increase and would have brought the server to its knees.

This brings us to one of my golden aims for reporting: "For the general case, an on-demand report (one that the user executes and actively waits for to render, rather than a report that's executed and received later) that cannot be displayed to the user within 10 seconds of the user asking for it is probably a badly designed report." As I discuss elsewhere in this book, I know that users get impatient. In some cultures, folks can get pretty irritated after a 10-second wait. Sure, in parts of Arkansas, it might take several hours for them to get a bit miffed, but in New York, watch out! Don't be fooled by performance that you see on a development system where you are the only user running a report. That's not a good benchmark for in-production performance.

Build a Custom Report Cache Table

In this situation, my data is relatively static, so I can help the situation by creating an output table that mirrors the dataset that the report requires and repopulating it on a regular schedule, perhaps an overnight job. In this case, I'm essentially caching the data so I don't have to endure the seemingly endless wait for the data to be retrieved. The table definition for the report table is shown in Figure 14.60.

Figure 14.60. The BiblioOrderStatusTimeAnalysis table definition.

[View full size image]

I can now populate this table by executing the T-SQL shown in Figure 14.61.

Figure 14.61. Populating the report data cache table.

[View full size image]

This approach enables us to issue very simple SELECT statements against the populated table from within the reportwithout fear of consuming vast amounts of CPU overhead on the server. The example in Figure 14.62 illustrates how this T-SQL is coded:

Figure 14.62. Returning data from the report cache table.

The data returned from this is still the 76,845 rows, but as all the heavy lifting and processing has been done beforehand, this resultset returns within 3 secondsthe CPU hardly registers any significant blip, since all it has to do is stream the data. However, in the pursuit of optimization, I created an issue of managing the updating of refreshing the data and updating this table.

Now that I have dabbled with how to optimize the data for a report, let's get back to the fun part of designing a report. Before long, you'll be back to data when you start to create parameterized stored procedures for the report to execute.

Building the Matrix Report

As I have done so many times in this chapter, I'm going to create a new project and report to host the Matrix report item. I provide only the basic instructions hereyou already know how to do most of these steps.

1.
Create a new Visual Studio Windows Forms project called BiblioAnalysis to host the report. Create a new report called BiblioAnalysisReport.rdlc.

At this point, I have to define how I am going to get the data. This requires adding a new data source to the project.

2.
Select Add New Data Source from the Data menu, and use the Data Source Configuration Wizard to build a TableAdapter against the Biblio BiblioOrderStateTimeAnalysis table I just created. Select all of the columns in the table for your TableAdapter. You'll also want to make sure the data is sorted by year, month, week, and day of the week (by number). Your query should look like Figure 14.63.

Figure 14.63. SELECT query to return rows from the BiblioOrderStateTimeAnalysis table.

[View full size image]

Tip

Don't forget to include an Application Name if you want to trace queries with the SQL Server Profile tool.

Designing the Report Layout Using the Matrix Report Item

I'm going to start the sample report using a Matrix report item.

1.
Open the report in the Report Designer and drag the Matrix report item to the report surface. Stretch it out a bit to make some space, as shown in Figure 14.64.

Figure 14.64. Configuring the Matrix report item.

2.
Next, select a Rectangle report item from the toolbox and drag and drop it into the area of the Matrix that says "Data" in a ghosted font.

3.
Return to the Data Sources window and drag the OrderValue and the OrderCount fields directly into the data Rectangle. You'll see that the Designer automagically creates TextBoxes for you and also creates aggregates of Sum on the field values. The designer did this because you are in the Data area of a Matrix, and the way in which a Matrix works only makes sense for aggregate values to be placed here.

4.
Return to the Toolbox and drag two more TextBox report items to the Matrix report item "Data" area to use as labels for the OrderValue and OrderCount TextBox report items. You'll need to fill in the text, as shown in Figure 14.65.

Figure 14.65. Positioning the OrderValue and OrderCount fields on the Matrix report.

5.
Drag the DayName field and drop it into the "Columns" area of the Matrix report item.

6.
Drag the WeekNumber field and drop it into the "Rows" area of the Matrix report item.

7.
Select the TextBox report items that contain data fields and set the Format property for the OrderValue and OrderCount. Use format expressions of "$#,###.00" and "#,###" for these two controls. This instructs the RDLC to format the money values so they are easier to read.

8.
Set the DayName TextBox report item to bold with a gray BackgroundColor.

9.
You'll find it helpful to select the Matrix and set the border property of each column and row to "solid". Drag the handles on the Matrix to tidy it up so that it is neatly presented, as shown in Figure 14.66.

Figure 14.66. The suggested layout of the Matrix report item.

It's important to make these cells as small as possible, as a Matrix report can get very large when the generated columns are added at runtime. To this end, I subsequently reduced the size of the "label" TextBox report items and shortened the text. I also manually set the "height" of these items to reduce the inter-cell spacing.

I also added a title TextBox and changed the BackgroundColor properties of the elements to make the report easier to read. This illustrates how the colors bleed through to the lower layers if you leave them set to "Transparent" BackgroundColor.

Testing the Matrix Report

At this point, we're ready to test the report to see if we've missed anything.

1.
Close and save the BiblioAnalysisReport.rdlc file and return to the Windows Form. Add a ReportViewer control to your form and click the ReportViewer Tasks button. Make sure that the BiblioAnalysisReport.rdlc is selected. This adds code to the Form_Load event handler to populate the TableAdapter and render the report.

2.
Verify that the Data Source has been wired to your report by clicking on the "Choose Data Sources" hyperlink. This launches the "Choose Data Sources Dialog". If the Data Source instance is not filled in, you'll need to drill down into the Project Data Sources and select the BiblioDataSet. Click OK to close the dialog.

3.
By using this technique to set the Data Source, Visual Studio and the ReportViewer create three items in the project tray. These classes (as discussed earlier in the book) expose the DataSet, BindingSource, and TableAdapter to your Form. Unfortunately, I don't want the report to render when the Form loads, so I'll move the generated code to my own button Click event, as shown in Figure 14.67.

Figure 14.67. Move the generated code to the button Click event.

[View full size image]

4.
Run the application and click the "Get Data" button to render the report. If you set the zoom out far enough, you'll see the entire reportat least, the first few weeks, as shown in Figure 14.68.

Figure 14.68. The Biblio Order Analysis Matrix report item report.

[View full size image]

The report shown in Figure 14.68 is a first step. If you navigate down to the third page, you'll see that there are only 53 rowsone for each week. However, the values are aggregated for all years, so it would be helpful to split out each year. One approach would be to drag the Year field to the Matrix rows Landing Zone, as shown in Figure 14.69. I also added text to the heading to reflect this added level of grouping. Again, these fields are made as small as possible to keep the report compact.

Figure 14.69. Adding another level of grouping to the Matrix.

I also made another change. As discussed earlier in this chapter, I wanted to be able to give the user the ability to see both versions of the report (with and without year aggregation), so I included code to switch from one report to another. Since both reports use the same Data Source, I added logic to bypass the Fill method once it's been done once. The code used to manage this report-switch approach is shown in Figure 14.70.

Figure 14.70. Switching between reports.

[View full size image]

When this new version of the application is executed, each year is broken out into a separate group and the report grows to about 20 pages, as shown in Figure 14.71.

Figure 14.71. The Biblio Order Analysis by Year Matrix report.

[View full size image]

Another technique that can be implemented with the Matrix report item is the "Calendar" effect. In this case, I use two nested List report items and nest the Matrix within the inner List item. To implement, this follow these steps:

1.
Select the Matrix report item (not one of the items contained therein) and open its extended properties dialog by right-clicking on the report item.

2.
Next, select the Groups tab and remove the Matrix1_Year rows grouping by selecting it and clicking Delete. Your Groups configuration should now look like Figure 14.72. Close the properties dialog.

Figure 14.72. The edited Matrix properties groups.

3.
Next, drag a List report item onto the Report layout; then, select the Year field from the BiblioDataSet and drag that into this List.

4.
Drag another List report item to be a child within the first List item. Place the MonthName field from the BiblioDataSet within this inner List item.

5.
Drag the Matrix report item to the inner List item. Figure 14.73 shows what the layout should now look like. No, we're not quite ready to test yet.

Figure 14.73. The Biblio Order Analysis Calendar Matrix report layout.

6.
Select the outer list and pull up its properties (right-click and select Properties). Click on "Edit Details Group". You should now see the "General" tab of the Grouping and Sorting Properties.

I need to group and sort the data on Year in descending order so that the report shows the most recent data values first. Using the drop-down list in the Expression dialog, choose "=Fields!Year.Value".

7.
Using this same Properties dialog, set the "Document map label" to "=Fields!Year.Value" (as shown in Figure 14.74) so the report displays a Document Map on demand. As you'll see, the document map facilitates navigation within the report. Click OK on the Grouping and Sorting Properties dialog.

Figure 14.74. Configuring the List item properties to group by year.

8.
I now need to set how the data is sorted. This needs to be done not on the Sorting tab of the Grouping property page, but on the Sorting tab of the List properties itself (as shown in Figure 14.75). Select the Sorting tab and, using the drop-down list of Fields, choose the Year and "Descending" as the direction.

Figure 14.75. Setting the List item sort sequence.

9.
Now, select the inner List and open its properties dialog (as shown in Figure 14.76). You may find that you need to set the Data Set name on General tab of the properties dialog. Again, simply use the drop-down list to select BiblioDataSet_BiblioOrderStateTimeAnalysisit's the only choice, as only one Data Source is defined for this project.

Figure 14.76. Setting the Data set name for the List item.

10.
Next, click the "Edit details group..." button. Set the Group on expression to the MonthName. Set the Document map label expression to the same field. Be sure to check the Page break at end, as shown in Figure 14.77. Click OK on the Grouping and Sorting Properties dialog.

Figure 14.77. Setting the inner List item properties.

11.
Click on the Sorting tab of the outer List item. In this dialog, set the Sort Expression to sort on the MonthNumber, as shown in Figure 14.78.

Figure 14.78. Setting the sort order on the inner List item.

12.
We're almost donejust a couple more steps to make the report look more like a normal calendar. If you run the report at this point, you'll get over 130 pages. If you look closely at the report, you'll notice that although the days of the week happen to be sequential, they don't have a consistent starting point from month to month. This can be corrected by opening the Matrix report item properties dialog as shown in Figure 14.72.

13.
Select the column grouping of Matrix1_DayName and click the "Edit..." button.

14.
In the Grouping and Sorting Properties for Matrix1_DayName, click the Sorting tab and select the "Sort on:" expression to the DayNumber Field value, shown in Figure 14.79.

Figure 14.79. Set the DayName sort sequence

15.
I encourage you to experiment with your report and make whatever changes you find useful. I made a few more tweaks and ended up with the report in Figure 14.80.

Figure 14.80. The final Biblio Order Analysis Calendar report.

[View full size image]

You can see the Document map on the left showing the Year numbers from the outer list in descending order, and then the month names from the inner list sorted in Month Number order. I added some code to tune the width of the Document Map area, but you can hard-code this by setting the DocumentMapWidth and DocumentMapCollapsed properties in the properties dialog.


Summary

This chapter provided you with an in-depth introduction to the ReportViewer control and the newest application paradigm exposed by Visual Studio. You saw that while the drag-and-drop techniques make Data Source generation easy, they aren't necessary in every case and sometimes don't afford the flexibility of hand-rolled queries. I showed you how to create your own DataTable and bind it to the ReportViewer LocalReport instance, as well as a variety of ways to launch your custom reports. This chapter should have made it clear that while the ReportViewer control exposes additional application report-generation alternatives, it comes at a price. With the ReportViewer control, you're in charge of far more functionality that has to be hard-coded and supported. The Reporting Services approach is less flexible with fewer options but makes it far easier for you and your team to build, deploy and maintain reports. If you need a more in-depth treatment of Reporting Services, be sure to read our book on SQL Server Reporting Servicesthis version of Reporting Services was developed for the SQL Server 2005 version, so the book really documents most of the current version. Sure, there are some improvements in the 2005 version, but I think you'll find that our book covers (or predicts) most of those enhancements.


Chapter 15. Summary and Wintry: Where We Are Now

This book provides a fairly broad treatment of Visual Studio and SQL Server. That's been my specialty for some time. When working at Microsoft University and over the last 20 years interacting with developers and other resources, I accumulated a wealth of information that's nowhere to be found in the documentation. When MSU folded, I moved to the Visual Basic documentation team where I was "permitted" to document Visual Basic but not SQL Serverthe documentation had to be "generic" and apply equally to all database backends. My books have tried to fill this gap between the language (Visual Basic) and the various versions of SQL Server.

In all of my books, I've tried to provide up-to-the-moment information, but this is really an impossible task, as Microsoft is constantly updating, fixing, and tuning the released products. I don't see an end to this constant cycle of early release, repair, and planned obsolescence as Microsoft attempts to maximize their profits by forcing development shops (and users) to buy their latest whiz-bang versions. It seems there is never enough time to get a product done right the first time and (sadly) never enough time to fix it once it ships.

So what have you learned from this book and its examples? That's a question only you can answer. What I've tried to do by writing this book is to make your work (and life) as a developer easier by passing on what I and others have learned over the years from working with and managing developers and other technical resources. No, I'm not the only expert out there, as well you know. There are many, many people that want to share what they know in an effort to help othersor simply try to make a buck. And yes, I have included their insights here as wellat least, the tips and techniques I endorse. I encourage you to continue to research and dig into other resources in your field and constantly hunt for other respected sources of information.

By experience, we've all found that Microsoft is not always (and, lately, not usually) the best source of information on its own products. It's ironic that in order to discover how a new Microsoft product or tool works, one has to find sources other than the Microsoft documentation. As a result, you and I spend countless hours scouring the Internet for the details of the basic functionality of the Framework(s) and the tools used to build Framework applications. We've also have to wade through far too many dated articles that apply to obsolete or beta versions. As a result, I asked Microsoft, SQL Server Magazine, and other folks that publish my work to purge their libraries of these dated articles. We've also found articles whose author is either ignorant, misguided, or misinformed, or simply has an alternate agenda. Knowing which articles contain valid, peer-reviewed, and technically sound information and which do not is not always easy.

I spoke to a senior staff member on the Microsoft Learning teamthe people responsible for writing the documentationasking why the help topics are in their current (pitiful) shape. She said their staff had been cut every quarter for the last few years and more cuts are planned. Microsoft had outsourced some of the work and was considering outsourcing more. Clearly, this strategy is not working. I certainly hope this trend can be reversedespecially since the list of new "features" being added to the technology keeps growing.

Just as clearly, Microsoft is not done. They seem to believe that it's their responsibility to constantly innovate and "update" the tools and platforms, servers, and infrastructure they expect you to use to design, code, debug, test, and deploy the applications you build. This is being done despite the reality that they don't have the people or resources needed to adequately test or document it. I'm of the opinion that there are just so many ways one can improve a hammer. So, many of our applications are fairly simple and need simpler, more easily understood tools to build them. We don't necessarily need all of the features they keep adding to their products. It seems that these easy-to-use "RAD" tools have gone the way of Visual Basic 6.0. Visual Studio and the Framework have grown progressively more complex. Given the upcoming changes to the fundamental way that applications are built (the Windows Foundation Classes), all I can see in the future is more and more "disruption in service" (as Ray Ozzie warned the attendees at TechEd Boston in June 2006).

In the coming months and years, I expect to see a fairly radical change in how Microsoft wants developers to address data access issues. We'll see DLINQ and Entity Data Modeling or EDM (another term for Object Spaces or Object Relational Modeling) make waves and attempt to win the interest of the development community. I don't plan to start writing in earnest about this technology until Microsoft decides to ship itbetter yet, consume it themselves. This means I encourage Microsoft to actually implement something (serious) with EDM or their new technology, and take it to the field as a real application before telling the world that Microsoft is ready for them to bet their own companies on the success and viability of the new and improved versions of the Framework, SQL Server, and Visual Studio.

If I've learned anything over the last 20 years working at or with Microsoft, it is that they are not going to stop changing as much as they can as quickly as they can. They seem to think it makes good business sense. So where does that leave you the developer trying to keep your skills up to date? Can you no longer depend on authors writing printed books? Perhaps not. As for me, I plan to switch my focus to EBooks (as I mentioned in the Introduction)at least for the foreseeable future. By publishing EBooks (or simply writing them), I think I can provide my insights and tips more quickly than if I have to wait for the paper books to grind their way out to your local bookstores. Frankly, this is an experiment. If I find that those who don't respect others' intellectual property (IP) rights tend to steal my content faster that I can write it, I'll find another way. Remember, much of the success of EBooks and the benefits they can provide hinges on you. If you seek out and download illegal copies of a book (or any media), you're contributing to the problem. Yes, I would really like to know if you think EBooks are a good idea or if you find someone stealing content. I also encourage you to provide feedback on this book and take advantage of the website setup to support it. Be sure to visit www.hitchhikerguides.net to ask questions, provide feedback, and interact with me and the others that provide support in that forum.


Appendix I. Installing the Examples and Test Databases

This appendix is designed to help you step through the process of installing the example applications included on the book's DVD. Generally, we're talking about two installation segmentsinstalling the examples and installing the sample databases. Before getting started, check for a "ReadMe" file on the DVD. It will have the latest updates to these instructions. Let's start with the examples.


Installing the Examples

The examples are located on the DVD in the Examples folder. These can be installed anywhere on your local system. No, because of "security" issues with Visual Studio, I don't suggest installing these on a network share. Simply copy the Examples folder to your hard drive. The examples are organized by chapter name, as shown in Figure I.1.

Figure I.1. The Examples folder on the companion DVD.

Previous Page
Next Page


Installing the Example Databases

The process of installing the example databases is a bit more complicated. Before getting started, you're going to want to choose a SQL Server instance to host one or all of the example databasesthere are six of them in the Databases folder on the DVD, as shown in Figure I.2.

Figure I.2. The "Databases" directory on the companion DVD.

No, you won't be able to attach the database files to SQL Server while they're on the DVDyou'll have to copy them to the system that's hosting your target SQL Server instance. Check with your DBA or whomever is managing your SQL Server instancesor look in the mirror and ask yourself if you have room for these databases on an existing instance. Perhaps, you should install a separate SQL Server instance to use as a test database. SQL Server Express is included with all versions of Visual Studioany or all of the databases and examples can be executed against a stand-alone SQL Server instance. I don't recommend using Visual Studio Express and simply adding any of these databases to your projects.

No matter which version of SQL Server you choose to use, you'll still have to configure it to support CLR procedures (if this makes sense to your DBA) and network visibility, as discussed in the book. I don't suggest that you install these sample databases on a production server or one that can't tolerate the extra load.

Tip

Don't install these sample databases on a production server or one that can't tolerate the extra load.

Attaching the Example Databases

Once the .MDF and .LDF files for each database you wish to install are copied to a (nonremovable) drive on the host system, you're ready to attach the databases. Let's take a quick look at the databases included on the DVD and listed in Figure I.2.

  • The Data directory This holds a set of flat files and other sample (non-SQL Server) databases.

  • BCPTest Used as a target by the SqlBulkCopy example.

  • BookSource Used in a number of the SQL Server CLR executable stored procedure and function examples.

  • Common Used to hold "common" SQL Server CLR executable functions.

  • Currency Used in a number of the SQL Server CLR executable user-defined type (typICurrency) examples.

  • GPS Used in the SQL Server CLR executables that access city/town lists stored with their GPS locations.

  • NewBiblio_Data (Biblio) This is the primary example database. It's used by the bulk of the examples in the book.

I suggest installing just the NewBiblio_Data.MDF (Biblio) database until you're ready to work with the more advanced portions of the book. While I've included a SQL script to attach all of these databases on the DVD, I suggest you use SQL Server Management Studio (or the Express edition) to attach the selected databases. While I discussed how to do this in Chapter 2, "How Does SQL Server Work?", remember that you simply right-click on "Databases" in the Object Explorer and choose "Attach". The dialog prompts you for an .MDF file location. SQL Server Management Studio automatically searches for the .LDF file and attaches both to SQL Server when you click "OK."

Establishing an Alias to Your Instance

All of the example applications reference the target SQL Server via an Alias of "DemoServer". This permits you to point to any named instance of SQL Server in the ConnectionString by using another namethus, "Alias". Creating an alias is very simplejust follow along:

1.
Start Microsoft SQL Server 2005 | Configuration Tools | SQL Server Configuration Manager. This MMC console application was discussed in detail in Chapter 2. It's used to manage the installed instances of SQL Server 2005, as well as establish aliases to any SQL Server instance.

2.
Click the "+" sign next to "SQL Server Native Client Configuration," as shown in Figure I.3 to expose the Aliases tree. Right-click on "Aliases" to expose the operations menu and choose "New Alias..." to create a new Alias.

Figure I.3. Opening the Aliases tree in SQL Server Configuration Manager.

3.
This opens the dialog shown in Figure I.4, which prompts you for the Alias Name. In this case, set this to "DemoServer"; the Port No, which I left blank to use an automatically generated port number; the Protocol, which defaults to "TCP/IP" (which you can set as requiredjust make sure the SQL Server has the selected protocol enabled); and finally the Server entry. The Server entry requires that you name the server as you would in the ConnectionString. This can refer to a "local" instance of SQL Server or a remote server hosted on another system on your network. See Chapter 9, "Getting Connected," for details on creating a valid SQL Server instance address. When you click "OK" in this dialog, the Alias is registered with the local system, so any references to the alias are translated to access the "Sever" entry.

Figure I.4. Configuring a new Alias.

Previous Page
Next Page


Chapter-Specific Configuration Issues

Some examples are a bit more difficult to use than others, due to issues with Visual Studio.

Chapter 8: Getting Started with ADO.NET

A few of the examples in this chapter access flat files, JET databases, and other ancillary files. These should be copied to a convenient place on your local hard disk or a network share. In the "ConnectionString Samples" example, the text files were placed in "D:\HHG\DVD\Databases\Data\Text Files". Of course, you can place these anywhere if you change the "strRootPath" string constant on line 4. You will also need to ensure that there is an ODBC-registered data source installed to access the text files. Use the Control Panel | Administrative Tools | Data Sources (ODBC) applet to configure a "Text File" system DSN.

Chapter 13: Managing CLR Executables

Because of an issue in Visual Studio 2005, one cannot correctly reference an Alias to the connection stringat least, not initially. For some reason, Visual Studio dereferences the Alias, looks up the actual server/instance name, and stores that behind the scenes. This means the first time you reference the alias, Visual Studio won't find it. This also means that when the Visual Studio solution file is first loaded, you'll have to point to an existing Data ConnectionVisual Studio will prompt you for that connection. This dialog also permits you to create your own connection. Since the CLR projects use one of several sample databases, I suggest that you click "No" when initially prompted to assign a new database connection for these CLR projects, as you'll want to know which of the test databases is being used. Open the project properties explorer and open the "Database" tab. Use the ellipsis button to choose an appropriate Data Connection, and set the initial catalog based on what's currently in the ConnectionString shown in the Properties dialog.


Summary

This appendix stepped you through the process of installing the code examples and example databases. If you have problems or questions, be sure to look for a "ReadMe" file on the DVD and then consult the support website (www.hitchhikerguides.net) for more help.


Appendix II. Reinstalling the DACW and Other Missing Functionality in Visual Studio

Microsoft has taken a moralizing stance on some of the new data access features. For reasons only they understand, they have chosen to "hide" some Visual Studio features that many of you have been using from the beginning. For example, every single ADO.NET book, article, or course discusses how to create an updatable DataSet with the DataAdapter Configuration Wizard (the DACW). If you start up Visual Studio 2005, you won't find this wizard in the Toolbox (at least, not in the Whidbey Beta 2 build). The "Command" and "Connection" wizard icons for each of the providers are gone, too. Thankfully, these are still in the product, but they've been hidden because Microsoft doesn't want you to use themor cut yourself while running. If you're like me, I still find uses for these productivity aidsespecially when teaching folks how to use ADO.NET. Yes, I expect there are still lots of developers who still depend on them and will miss them as soon as they start using Visual Studio 2005. Thankfully, it's not hard to add them backjust follow these steps:

1.
Start Visual Studio and create a Windows Forms project. Sure, this can be done in any languageeven C#.

2.
Pin down the Toolbox tab, as you're going to create a new tab.

3.
Right-click below the items in the Toolbox and click "Add Tab". Give your new tab a meaningful name, like "Deprecated Data Components That Microsoft Doesn't Want Me to Use".

4.
Right-click again in your new tab and select "Choose Items...". This exposes the "Choose Toolbox Items" dialog (after a long wait while it's initialized).

5.
Scroll down to the section for your data access provider. For SQL Server, you'll want to add items from the list of SqlClient components. If you type "S", the pointer will move promptly to the first "S". You can also enter "SqlClient" in the Filter dialog to locate all matching components (as shown in Figure II.1).

Figure II.1. Adding back deprecated components in the Toolbar.

6.
Repeat this process for the ODBC and OLE DB components that you want to expose in the Toolbar. When you're done, you should have a new Toolbox tab with the ADO.NET components you've used for the last three versions (as shown in Figure II.2).

Figure II.2. The Toolbox with the missing ADO.NET 1.1 components replaced.

Unfortunately, not all of the productivity features we got used to in Visual Studio 2003 are present in Visual Studio 2005 and can't be added back. For example, when you want to create a quick call to a stored procedure and build the Parameters collection, you can (in Visual Studio 2003) drag a stored procedure from the Server Explorer to the form, and Visual Studio takes care of the rest. Sadly, this does not work in the 2005 version. I understand that one of the new refactoring add-ins can build a Parameters collection from a stored procedure, so all hope is not lost.


Appendix III. Monitoring SQL Server

One of the reasons SQL Server has become so popular is that it's so easy to monitor. From my very first Hitchhiker's Guide, I've been able to use the SQL Profiler to see what hoops the data access interfaces are asking SQL Server to jump through. In the past, Microsoft's data access strategy was built around a number of property/method-driven "black boxes" that drove the backend-specific data drivers or providers. Visual Basic, Visual Basic.NET, C#, and Visual Studio all have T-SQL code generators that were also hidden from the developer. SQL Server Management Studio and its toolsets are no exception. By monitoring the SQL Profiler, you can easily see each and every command given to SQL Server and get a real-time feel for the load it's under and what's generating the traffic. In addition, you can see what these black boxes are generating syntax-wise. This makes it easy to adapt your code to "help" the code generators create more efficient queries.

Of course, the SQL Profiler is not the only way to monitor SQL Server or even ADO.NET itself. The ADO.NET data provider, .NET Framework, and the SQL Server service all expose performance counters that can be monitored either by using the Windows Performance Monitor or by your code. Visual Studio exposes a Performance Monitor control that can make it less difficult (I can't say easy) to monitor these performance counters.

This appendix walks you through the process of setting up a basic SQL Profiler session, as well as shows you a simple application I wrote for one of my conference sessions that monitors the ADO.NET connection pool through use of the ADO.NET .NET data provider performance counters.


Monitoring SQL Server with the SQL Profiler

The SQL Profiler is not included with SQL Server Express or Visual Basic Express, so don't bother looking for it. It also can't monitor User Instances (at this point in time), so don't bother trying. It can, however, monitor both local and remote SQL Server service instancesall you need to do is open a connection to the target service. Let's walk through a typical SQL Server Profiler scenario. Suppose you have a new stored procedure you're trying to run, and you're wondering why it's getting a syntax error. Perhaps, your code or ADO.NET (like the CommandBuilder) is generating some of the code, and you wonder if it's confused (again).

Adding Filter Criteria

Before you run off and launch the Profiler to start scanning the stream of T-SQL being passed to your SQL Server instance, it's a really good idea to think about how you're going to pick your application's commands out of the flood of commands being sent to the server. Consider your instance of Visual Studio, Reporting Services, Notification Services, as well as every other application and instance of Visual Studio in your entire organization that's using the selected instance. SQL Server is constantly polling the server, running queries, and adding to the work SQL Server is asked to perform. Monitoring SQL Server with the Profiler is like trying to understand a scintillating conversation in a noisy baryou'll need a lot more than a few disjointed words to make sense of what your friend is saying. An easy way to "mark" the traffic your application generates is by setting the Application Name property in the ConnectionString when you open the connection. As I'll illustrate in a moment, SQL Profiler can create a trace that only reports on the activity from your application or any subset of the traffic. Without some kind of filter, you won't make much sense out of the data it presents.

Suppose you're trying to use Visual Studio to walk through a stored procedure from the Server Explorer. In this case, Visual Studio will use the current Data Connection's ConnectionString to open the connection. Here is where you need to set the Application Name property.

1.
Right-click on the Data Connection you're using in the Visual Studio Server Explorer and choose "Modify Connection". This opens the Modify Connection dialog.

2.
Click the "Advanced" button. This opens the Advanced Properties dialog, shown in Figure III.1. Scroll to the Context section and set the Application Name (or Workstation ID) property to a unique value. The default value for Application Name is set by Visual Studio to ".Net SqlClient Data Provider" so this same Application Name is used by every other Visual Studio application using a Data Connection to access SQL Serverthat's why you need to make the Application Name key unique, so you can pick out your traffic from the cacophony of commands being monitored.

Figure III.1. Setting the Application Name (or Workstation ID) in the Advanced Properties dialog.

3.
Click "OK" to accept the new ConnectionString for the selected Data Connection.

Configuring SQL Profiler

Now that the ConnectionString has a unique value that can be used to filter its traffic out of the stream of trace events, I'm ready to start tracing.

1.
Launch the SQL Profiler by using the Start | All Programs | SQL Server 2005 | Performance Tools | SQL Profiler. Yes, the Profiler is also supported in older versions of SQL Serverit's been around for some time. All of the functionality demonstrated here is in SQL Server 2000 as well as SQL Server 7.0 (to a great extent).

2.
At first, the SQL Profiler is simply sitting there waiting for you to start a new Trace. We're about to do that. Click the File | New Trace menu. This opens the "Connect to Server" dialog. Fill in the server name\instance for the instance of SQL Server referenced in your Visual Studio Data Connection's ConnectionString (from step 1). Choose appropriate authentication credentials to log on to the SQL Server instance.

3.
This opens the Trace Properties dialog, shown in Figure III.2. If you accept the default properties, you'll probably get the unfettered stream of commands I've been talking about, so let's set the needed filters to make sure you see only pertinent events. Click on the "Events Selection" tab.

Figure III.2. The Trace Properties dialog is shown when creating a new trace.

4.
This opens the Events Selection dialog, where you're presented with a dizzying array of options from which to choose. Feel free to experiment here, but we want to narrow our focus to setting the ApplicationName filter criteria, so click on that column, as shown in Figure III.3.

Figure III.3. The Trace Properties dialog Events Selection tab.

[View full size image]

5.
This opens yet another dialog that's used to set the criteria for the Application Name filter, shown in Figure III.4. Each of the other events and event columns has similar dialogs, so it's possible to set any number of filters on the data to be returned by the trace engine. Set the Like expression to test for your unique Application Name string. Notice that this is a Like expression, so you can use a wildcard (%) to make this easier. Click "OK" when you're done.

Figure III.4. Setting the Application Name event filter to test for your unique value.

[View full size image]

Starting the Trace

Once SQL Profiler has been configured to filter out other traffic, we're ready to start the trace. I think you'll find that even given a filter that shows only traffic from your unique connection, you're going to want to revisit the Trace Properties dialog Events Selection tab to further refine the criteriayou're still going to get more information than you need. That's because Visual Studio performs a staggering number of housekeeping and other functions to gather information about the SQL Server objects it's being asked to manipulate.

1.
Click "Run" on the Trace Properties dialog to start the trace. This opens the Trace View window (shown in Figure III.5), which is simply a real-time view of all trace events that meet the criteria you set in the Trace Properties dialog. You should see an "ExistingConnection" event in the window (in green). We're ready to return to the application to start testing our stored procedure.

Figure III.5. The Trace View window shows unexpunged stream events.

[View full size image]

2.
Return to Visual Studio and, using the Server Explorer, right-click on the stored procedure to test (I'm testing the CustomerUpdate procedure), and choose either "Execute Procedure" or "Step Into Procedure". This opens the Run Stored Procedure dialog, shown in Figure III.6.

Figure III.6. Visual Studio prompting for stored procedure input (and input/output) parameters.

3.
Click "OK" to accept the parameters. Sure, you can fill in the parameter values, but remember, this is an UPDATE stored procedure, so you'll be changing the data as you click "OK".

4.
Switch back to the SQL Profiler and observer the Trace View window, as shown in Figure III.5. This window is named based on the name you gave it in the Trace Properties dialog. Yes, you can have several traces running simultaneously. It should now show the T-SQL commands executed by Visual Studio when you executed or started to step through the stored procedure.

Evaluating the Trace

As you can see in Figure III.6, SQL Server was asked to execute about five queries to step through the stored procedure. Sure, it might make sense to filter out the "BatchStarting" or "BatchCompleting" events (but not both). This would cut down the volume of events by half. This can be done by unchecking the undesired event in the Trace Properties dialog (as shown in Figure III.3).

Note that Visual Studio (or SQL Server Management Studio) adds a lot of extra events to the streamit does quite a bit of work behind the scenes that (hopefully) won't appear on a production system where your applications are not competing with developers and Visual Studio queries. However, your SQL Server might have to support queries from FullText Search, Reporting Services, Analysis Services, and a host of other SQL Serverrelated services that can appear in the trace and can materially affect performance.

A Few SQL Profiler Tips

Here are a few tips and suggestions for optimal use I've learned over the years when working with the SQL Profiler:

  • A trace can be paused and restarted at any time, but restarting the trace flushes the Trace View window.

  • It makes a lot of sense to save your events filter configuration so that next time you start SQL Profiler, you can simply create a trace from an existing template.

  • Remember that the SQL Profiler consumes CPU resources and RAM, so leaving it running (with an active Trace) can hurt performance.

  • The SQL Profiler can be used to create a long-term trace log that can be fed back into SQL Server Management Studio tools to evaluate overall system performance and suggest changes in the indexes or other SQL Server configuration options. See BOL for more details.


Monitoring SQL Server and ADO.NET with Performance Counters

Okay, we're ready to check out another technique to monitor SQL Server and the SqlClient .NET Data Provider. Sure, you can also monitor about a bazillion other performance counters besides those exposed by SQL Server and ADO.NET. Unfortunately, working with performance counters can be problematic. First, there is an entire set of counters that should not be usedthose created for the 1.0 version of the frameworkbecause they never have worked correctly. I'll show those to you a bit later. Let's get started by using the Windows Performance Monitor to monitor activity on our selected SQL Server instance.

1.
Start the Windows Performance MMC snap-in from the Start | All Programs | Administrative Tools menu. This opens the Performance application, which, by default, starts to display the status of three performance counters: Memory Pages/Sec, PhysicalDisk Avg. Disk Queue Length, and Processor % Processor Time. A snapshot of these counters is taken once per second, and their value is drawn on three graphs shown in Figure III.7.

Figure III.7. The Performance monitor's default configuration.

[View full size image]

2.
After removing these three counters (by clicking on the "X" button at the top of the window for each counter to delete), I clicked on the "+" to open a selection dialog where I can choose other Performance counters to monitor. This dialog can take a few seconds to load, so be patient. Let's start by choosing the ".NET Data Provider for SqlServer" counters from the dropdown list, as shown in Figure III.8. Do not select the .NET CLR Data countersthey're busted and won't be fixed.

Figure III.8. Selecting a set of Performance counters.

3.
Okay, once you choose a set of counters, the next (fairly important challenge) is knowing which instance to choose. These "instances," shown in the right-hand ListBox in Figure III.8, are really connections through the SqlClient .NET data provider. No, if you don't have any active connections open, this list will be empty. For the moment, let's sidestep that issue. The Add Counters dialog should now look something like Figure III.9.

Figure III.9. Selecting a specific Performance counter to monitor.

4.
If you click "Add" at this point, you're telling the Performance application to start monitoring data from the selected Performance counter as generated by the selected instance(s). That's coolright up to the point where the connection is closed. That's when the SqlClient Performance counter code starts throwing exceptions each time its referencedonce a second, in this case. If the connection is re-established, the instance number changes as a new connection is created. This makes these SqlClient counters not as usefulbut there are more counters from which to choose. At least they don't affect the behavior of the connection pool like the old 1.0 counters.

5.
Browse through the list of exposed Performance counters shown in the ListBox on the left. All of these pertain to ADO.NET connectionsno other ADO.NET metrics are exposed. No, not all of the counters are exposed in this listand they won't be until you tell your ADO.NET (via the ConnectionString) that it's okay to return the values. That's because some of these counters (as I describe later) are very expensive to compute, so they can affect performance.

Before I drill into the SqlClient Performance counters, consider that SQL Server and its related services expose a wealth of Performance counters. These include one or more counters (usually a dozen or more) exposed by Integration Services (BCP/SSIS), Native XML Web Services, OLAP, Notification Services, Reporting Services, Replication Services, and a host of Performance counters exposed by SQL Server itselfas well as many others too numerous to mention[1] In this same vein, SQL Server General Statistics: Logical Connections is especially useful when monitoring the state of the connection pool.

[1] See "Using SQL Server ObjectsMonitoring Resource Usage (System Monitor)" in BOL.

The ADO.NET SqlClient Performance Counters

After the (ah, to be kind) debacle created by the 1.0 ADO.NET Performance counters, Microsoft decided to start over and create a whole new set of counters that return basically the same informationand a bit more for the 2.0 release. Let's hope these hold up better. Nope, these counter names are not localized, but duh, that would be really tough to support. I've listed these in Table III.1.

Table III.1. SqlClient ADO.NET Performance counters

Performance Counter

What It Counts...

HardConnectsPerSecond

Connections per second made to specified connection instance.

HardDisconnectsPerSecond

Connections per second made to specified connection instance.

NumberOfActiveConnectionPoolGroups

Unique active connection pool groups. Controlled by unique connection strings in the AppDomain.

NumberOfActiveConnectionPools

The total number of connection pools (see Chapter 9, "Getting Connected").

NumberOfInactiveConnectionPoolGroups

Unique connection pool groups marked for pruning. Controlled by unique connection strings in the AppDomain.

NumberOfInactiveConnectionPools

Inactive connection pools without recent activity and waiting to be disposed.

NumberOfNonPooledConnections

Active unpooled connections.

NumberOfPooledConnections

Active connections managed by the connection pooler.

NumberOfReclaimedConnections

Connections reclaimed through garbage collection where Close or Dispose not called by the application.

NumberOfStasisConnections

Connections currently awaiting completion of an action; therefore, unavailable for use.

SoftDisconnectsPerSecond

Active connections being returned to the connection pool.

*The following counters are not enabled by default. I show how to enable them in this section.

NumberOfActiveConnections*

Active connections.

NumberOfFreeConnections*

Connections available for use in connection pools.

SoftConnectsPerSecond*

Active connections pulled from the connection pool.

Activating "Special Case" Performance Counters

The aforementioned (*) counters consume more resources than the others, and the SqlClient provider won't populate them unless you add the following element to your application configuration file (as shown in Figure III.10).

Figure III.10. Add this element to your application's app.config file.

[View full size image]

When wiring up each of these counters, you'll need to provide a valid "instance" that's tied to the process ID of the application that "owns" the connection pool. Remember, each process or app domain (in an ASP application) gets its own connection pool. If the application ends, any applications monitoring the connection pool instance must be prepared for the inevitable exceptions.

If you do anything in any application that monitors Performance counters, you need to ensure that your exception handlers are ready for lost connectivity to the selected instances. I've included a sample application that monitors and exercises the connection pool and uses both SQL Server and SqlClient Performance counters. See "PoolMonitor Using Performance Counters" on the DVD. This application "owns" the connection pool it creates, which makes it easier to capture the instance namethe code in Figure III.11 shows how this is done.

Figure III.11. Capturing the ProcessID and determining the instance name of the connection pool.

[View full size image]


Summary

This appendix focused on two essential tools that developers use to monitor SQL Server. The SQL Profiler has helped me for over a decade to understand how the data access interfaces interact with SQL Server. As each new interface arrived, I was able to tell exactly what was working and what was not by monitoring the SQL Profiler traces.

Performance counters have been around for some time as well. Now that Microsoft has the ADO.NET connection pool counters working, these can play a pivotal role in ensuring that your application is not leaking connections.


Appendix IV. Creating and Managing Server-Side Cursors

They told you it couldn't be done. They were wrong. Yes, you can create and manage your own server-side, fully scrollable and updatable cursors on T-SQL Serverand you can do it all with ADO.NET. This appendix is excerpted from a popular magazine appendix that focused on missing functionality in the then-current version of ADO.NET. Since then, the problem of missing server-side cursors has not been fixedand it probably won't be. For a litany of reasons, Microsoft chose not to implement any server-side cursor functionality in any version of ADO.NET. This appendix discusses how you can work around this limitation to create and manage your own server-side cursors by using T-SQL ANSI T-SQL statements. Yes, this a lot more trouble than using ADO classic's (default) server-side cursor, but at least it's one viable (albeit a bit unusual) way to solve this problem.


Why Are Server-side Cursors Important?

For over a decade, T-SQL Server and other DBMS developers have been using server-side cursors to access their databases and scroll through updateable rowsets. A server-side cursor in its purest form is simply a block of pointers to live data rows as defined by a SELECT statement. The cursor can include all rows from a specific table or just a selected few. It can address rows generated from a JOIN, view, or stored procedure. Again, when you create a server-side cursor, you aren't asking for data; you're asking for pointers to the data. This means that the data remains where it should remainon the server, where other clients can see and change the values. When you fetch rows from a server-side cursor, you are fetching the "live" dataas it is on the server that instantnot some cached version that's been sitting around in memory for a few seconds to a few days.

Server-side cursors were implemented in the earliest versions of RDO and as the default cursor in ADO classic. This "connected" data access paradigm assumes that the database connection remains in place while the application runs a query and builds a server-side structure that permits one to browse up and back through a set of rows that can be retrieved and updated as needed.

Server-side cursors are especially useful when working with highly interactive applicationsespecially when the application cannot work with disconnected (static) data. This type of application needs a mechanism to work with a single row or a small set of rows at once. Server-side cursors are designed to meet this need. As illustrated by the examples in this appendix, I'll show you how to:

  • Create a cursor based on a focused SELECT statement

  • Position a cursor to any designated row

  • Change the data in the currently selected cursor row

  • Adapt the cursor so other users' changes are visible (or not)


How Does ADO.NET Implement Cursors?

ADO.NET is fully capable of running queries, but the "cursor" it knows how to create is not the same as DAO, RDO, or ADO "classic" (ADOc) developers might be used to. In the strictest sense, a cursor is a device used to browse through the rows that result from a query. Some cursors are scrollable, so you can position a "current-row" pointer at arbitrary points in the rowset and return a specific row, while others are "forward-only". I expect most of you are interested in fully scrollable cursors.

Yes, if you squint a DataTable might be thought of as a cursor, in that it exposes a Rows collection that can be browsed (or scrolled) in any order. But a DataTable is static client-side datait's really no different than simply taking a DB-Library data stream and dumping it into a local array. While server-side cursors can also be static data (that does not change as the database is updated), server-side cursors can also be a fixed set of keys (a keyset cursor) or a dynamically changing set of keys (a dynamic cursor) that points to live data. That's the critical difference. Server-side cursors leave the data on the server (or they can), returning only "pointers" in the form of keys that point back to live data. As other users change the data, the cursor member keys still point to fresh data.

The ADO.NET architects and Microsoft product managers think that static, client-side DataTable and TableAdapter objects are enough for many customers. They're rightup to a point. I think that quite a few applications designed to use server-side cursors could have been written using client-side static cursors. However, when they are, developers are sometimes forced to make extra round-trips to the server to detect changes in the data or membership. Another approach Microsoft is promoting is T-SQL Server Notification Services, where you can set up call-backs to your application when specifically (and carefully) selected data changes. However, this approach is not for the faint of heart or simple, fast, mainstream applications.

I think there are plenty of viable situations where a server-side cursor makes sense. I keep hearing that Microsoft was concerned that developers were "misusing" server-side cursors. That's understandable, as the default cursor library (CursorType) in ADO implements a server-side cursor (adUseServer)so it's their own fault. In addition, too many examples illustrated fundamental operations using (often expensive) server-side cursors. I think that developers are smart enough to make up their own minds, and server-side cursors can increase developer and code performance if used wisely.

This approach to data access is not particularly interesting in stateless Web-based applications. It makes the most sense in connected Windows Forms applications.


How Are Server-Side Cursors Managed?

All of my Hitchhiker's Guide books have discussed cursors at lengthat least the ones where ADO, RDO, or DAO implemented them. Since ADO.NET does not support this paradigm, I'll need to add a few paragraphs here to refresh your understanding of the mechanics and terminology.

First, when you execute a query to build a server-side cursor (or any cursor, for that matter), SQL Server looks through the database to find qualifying rows. Your WHERE clause and other SELECT statement operators determine which rows are to be included and which are not. While this operation is not instantaneous, the rows found become part of the cursor "membership." If there is no WHERE clause in the SELECT query, all rows in the target table(s) are made members of the cursorthat can be expensive. Remember that server-side cursors (like a keyset cursor) are simply a temporary collection of pointers to the actual rows.

There are several types of server-side cursors[1]:

[1] See "DECLARE CURSOR" (Transact SQL) in BOL.

  • Static cursors contain a scrollable set of rowsnot just keys. SQL Server creates a one-time temporary copy of the data in TempDb, so membership is fixed. References to the cursor are fetched from this temporary table. Static cursors are not updateable.

  • Keyset cursors contain a set of keys used to fetch the actual rows. SQL Server creates a one-time temporary table that contains PKs of member rows, so membership is fixed. References to cursor rows are fetched using keys from this temporary table. As changes are made to the database (to non-key columns), these values are made visible when the row is fetched (or refetched). If a member row is deleted, @@FetchStatus returns 2. Rows addressed by keyset cursors are updateable through use of the UPDATE WHERE CURRENT OF syntax (that's illustrated in the example application). You can position to any specific member row using the ABSOLUTE fetch option or scroll through rows using FETCH NEXT, PRIOR, LAST, and FIRST fetch options. Keyset cursors default to SCROLL (scrollable) positioning but can be set to FORWARD_ONLYhowever, that does not make much sense to me.

  • Dynamic cursors are like keyset cursors in most respects, except that membership is dynamicthat's because each time a Dynamic cursor row is accessed, the SELECT query is re-executed to repopulate the membership. These cursors are expensive and do not support absolute positioning.

  • Forward-Only is an option for the other cursor types that disables the ability to scroll through the rows. FETCH NEXT is the only available fetch option for FORWARD_ONLY cursors. When declared without the STATIC, KEYSET, or DYNAMIC keywords, the FORWARD_ONLY cursor defaults to DYNAMIC (which means that you should always indicate one of the other types).

Other cursor options permit you to specify if the cursor can be "seen" by other applications accessing SQL Serverbeyond your application's connection. See SQL Server 2005 BOL for more specific details on these options. They describe how to make cursors visible to stored procedures and elsewhere.

What Do Server-Side Cursors Cost?

One of the downsides of server-side cursors is the burden they impose on T-SQL Server and the scalability of your applications. When you create a server-side cursor, T-SQL Server writes a set of keys or entire rows to memory or TempDB. This process is repeated and its effects multiplied for each application that opens a server-side cursor. Using server-side cursors also assumes that the connection remains open as long as the cursor's data is needed. This is definitely not a good idea for a Web-based (ASP/ASPX) application because the connection is usually closed, and its resources released, just after its queries are executed. On the other hand, for many traditional client/server applications, server-side cursors have been an effective way to manage data and simplify application designwhen used wisely.


How Can ADO.NET Create a Server-Side Cursor?

In my ADO.NET book, I state (repeatedly) that ADO.NET cannot create a server-side cursor. Strictly speaking, this is trueon its own, it can't. However, it's true only in the sense that it can't create and manage a server-side cursor on its own in the same way that DAO and ADOc can. As I'll show you in this appendix, you can create server-side cursors using some fairly easy ADO.NET methods.

It's not a good idea to use the following techniques when working with ADOc. That's because the server-side cursor mechanisms within ADOc can conflict with any cursors you create on your own using T-SQL.

This technique is really pretty simpleit's based on using T-SQL to create ANSI cursors. As illustrated in the example code, I use the ADO.NET Command object to execute specific T-SQL scripts (two or more tightly coupled T-SQL statements), including the DECLARE CURSOR statement and the FETCH and DEALLOCATE ANSI cursor commands. Let's step through the process of creating and managing a cursor.

Opening, Closing, and Re-opening the Connection

Unlike other ADO.NET examples in this book and elsewhere, when building a server-side cursor, the code needs to open a connection to the T-SQL Server instance and keep it open. That's because the cursor is built and maintained by the server and owned by the connection. Once the connection closes, the cursor is (usually) flushed[2] (at least when the connection pooler closes or resets the connection).

[2] It's possible to create a database-persistent cursor using the GLOBAL option.

As discussed in Chapter 9, "Getting Connected," consider that a Windows Forms application creates a new connection pool for each unique connection string. This means that if you create a cursor, it won't be destroyed (at least, not immediately) when you close the connectionassuming that you're using connection pooling (which is on by default). However, if the connection reset option is enabled (it is by default), the cursor is cleared when the connection is reused again so the new connection owner cannot inherit the cursor. If you turn off the reset connection option, the cursor should remain viable for reuse by your applicationit will still be there when you reopen the connection. However, if the connection pool times out the connection, which forces the connection closed, your cursor is lost. Of course, in a Windows Forms application, it's perfectly reasonable to disable connection pooling. In this case, when the connection is closed, the cursor is lost. No, the data addressed by the cursor is undisturbed when the cursor is closed.

Creating the Cursor

The next step is to create the cursor based on a T-SQL statement. This block of code uses the T-SQL statement passed from the user in a TextBox control. (No, it's not a good idea to just accept anything from a user, but this is a demonstration and test program, so I'm pretty generous with what I permit in the T-SQL.) To be safe(er), I call the ValidateSQL routine to do some rough tests to make sure I'm not too sleepy and enter something destructive.

The T-SQL used to create the ANSI server-side cursor can be broken down into three parts (as shown in Figure IV.1):

  1. The DECLARE cursor statement, which names the cursor and sets the options to use. In this case, the code asks for a simple scrollable cursor (as opposed to a forward-only cursor) and names it myCursor.

  2. The T-SQL SELECT statement, which is executed to build the cursor rowset, follows the CURSOR FOR argument. The SELECT statement should focus the cursor on a few manageable rows in the databasenot an entire database tableif you expect to scale the application. Sure, the SELECT can contain a JOIN or a VIEW, as long as it's a standard (simple) SELECT statement. The keywords COMPUTE, COMPUTE BY, FOR BROWSE, and INTO are not allowed.

  3. A SELECT query to return the number of rows in the cursor. If you select a large number of rows, this query will probably return 1 until T-SQL Server has completed server-side rowset population (until SQL Server has found all of the member rows and added them to the cursor).

Figure IV.1. Constructing a server-side cursor.

[View full size image]

Next, the code executes the T-SQL using the ADO.NET Command object's ExecuteScalar method. This returns the @@CURSOR_ROWS global variable, so I know how many rows are in the cursor. Remember not to close the connection after having created the cursorif you do, the cursor will be dropped by T-SQL Server.


Fetching Data from the Cursor

Once the cursor is open, you have lots of options. The data can be fetched once or any number of times and in any order (since the code created a scrollable cursor). Yes, as in ADO classic cursors, a "current position" must be maintained. But this time, it has to be handled in your codeADO.NET won't help here. You use the TSQL FETCH operator to retrieve a specific row (and only one row at a time) from the server-side cursor. The FETCH operator can be set to fetch the FIRST, LAST, NEXT, or PRIOR rows based on the current row pointer simply by using the appropriate operand. You can also fetch a specific row (the "nth" row) using the ABSOLUTE operand; or, if you want to fetch a row that's positioned in the cursor a specific number of rows forward or back, you can use the FETCH RELATIVE operand.

Let's go over these techniques and how they're implemented in code (as shown in Figure IV.2):

  • To fetch the "first" or "next" row: When the cursor is first opened, the "current row" pointer is positioned before the first row (just like in ADOc), so either FETCH FIRST or FETCH NEXT gets the first (or next) row as defined by the declared cursor SELECT statement. In this case, simply set the CommandText with the appropriate FETCH TSQL operator and execute it. Note that the code returns a second resultset that contains the @@FETCH_STATUS global variable. This tells the application if the code has fetched past the last row or before the first row, or if something else went wrong. If you position past either end of the cursor, @@FETCH_STATUS returns 1 and there is no "current" row.

  • To fetch the previous row: You can use FETCH PRIOR to step backward through the rows until @@FETCHSTATUS returns 1, which indicates that you've stepped over the "BOF" line.

  • To fetch a specific (absolute) row: Use FETCH ABSOLUTE with a specific row number to fetch. Note that cursor rows begin with 1 and end in @@CURSOR_ROWS.

Figure IV.2. Using the T-SQL FETCH operator to reposition the current-row pointer.

[View full size image]

Fetching the Entire Cursor

The T-SQL ANSI cursor implementation does not support the ability to fetch all of the rows in the cursor at once. This makes sense because once you fetch the rows into the client, they start getting stale the instant they arrive. However, I think fetching the entire cursor might be useful when you want to display the members of the cursor and let the user choose one of them to update (yes, it's possible to update through the cursor).

While it's possible to loop through multiple FETCH NEXT calls to get each row one at a time, it's far faster to get the rows in blocks. This means that instead of one round-trip for each of 100 cursor rows, the code executes only ten round-tripsfetching 10 rows at a timeassuming it uses a batch size of ten rows. Behind the scenes, the example code sets the number of FETCH operations to the number of expected rows in the cursor, but not greater than about 150 or so. Yes, you can get the number of rows in the cursor as soon as SQL Server completes membership population. Remember the SELECT @@CURSOR_ROWS T-SQL command?

To carry off this batch fetch approach, the example builds a long string to contain a set of FETCH NEXT operators, as shown in Figure IV.3. This is done using the StringBuilder class to reduce string handling overhead. In this case, sbFetchNext is declared as a new StringBuilder and intBatchSize is declared as 100. Remember, this routine simply calls FETCH NEXT N times so it starts from the current row. To have the routine fetch rows from the top, simply execute a FETCH FIRST before calling the FetchAll routine. Yes, if there are less than cursor rows MOD N, some of the FETCH NEXT commands will return empty rows. The example code is prepared for that.

Figure IV.3. Create the bulk fetch string.

[View full size image]

Executing this batch of FETCH operators causes T-SQL Server to return N+1 resultsetsone for each row in the batch and one to contain the @@FETCH_STATUS. I used a few new ADO.NET 2.0 methods and techniques to construct the output DataTable containing the rows returned from the FETCH NEXT statements, as shown in Figure IV.4. This routine has been completely re-engineered since first publishedall because of the new ADO.NET features that make it far, far faster than earlier versions.

Figure IV.4. Fetching the entire cursor from the current position.

[View full size image]


Updating with a Server-Side Cursor

While the example program written for this appendix does not attempt to update the cursor data, it's not that hard to do (though it does expose a much wider set of issuesfodder for another appendix). Suffice it to say that the cursor can be updated through use of the CURRENT OF expression in the UPDATE statement's WHERE clause. For example, to position to a specific row (based on the cursor row, 1 to @@CURSOR_ROWS) and update that row, you could code a routine like the one shown in Figure IV.5.

Figure IV.5. Executing the UPDATE statement using the WHERE CURRENT OF option.

[View full size image]


Summary

When I first started working with ADO.NET, I, like other developers, trainers, mentors, and architects, found that many of the techniques we've used for years in DAO, RDO, and ADO are no longer available. However, as we learn to adapt, it seems that easy work-arounds for some of our tried-and-true approaches have appeared. This appendix discussed one such work-around. In other cases, our well-worn techniques might not have been such good ideas in the first place.


Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]


SYMBOL

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

% (percent sign)
& (concatenation) operator
/* */ comment tags
@@IDENTITY global variable 2nd 3rd 4th
[] (brackets)
_ (underscore)


A

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

access rights for parameter-based queries
accessing
ADO classic ADTG Recordset
flat files
JETdatabases
Accumulate function
action commands
INSERT
Identity values
NULL and DEFAULT values
UPDATE
BLOBs
concurrency
decimal and floating point numbers
parameter-based updates
SET clause
strings
strongly typed data columns
activating
connection pools
performance counters
ad hoc queries, integrating into applications 2nd
adCriteriaAllCols setting (Update Criteria property)
adCriteriaKey setting (Update Criteria property)
adCriteriaTimeStamp setting (Update Criteria property)
adCriteriaUpdCols setting (Update Criteria property)
Add method 2nd
Add New Table dialog
AddingNew event
AddLoan stored procedure
AddRange method 2nd
addressing objects
AddTrustedCodeModuleInCurrentAppDomain method
administration
administrative functions
Auto Close option
backups
importing/exporting data
.MDFdatabase files, attaching
DBMSs
SQL Server Express Edition 2nd
ADO
ADTG Recordset, accessing
class diagrams
data sources
history of
overview of
ADO and ADO.NET Examples and Best Practices
ADO.NET
class names
CLR executables
Querying data
returning data/messages
SqlContext connection
transactions
CommandBuilder
concurrency management
example
data access providers
DataColumn objects
DataRow objects
DataSet objects
overview of 2nd
populating
strongly typed DataSets
DataTable objects
DataColumns
DataRows
joins
overview of 2nd
populating DataSets with
populating from SqlDataReader
DataView class
exploring with Visual Studio Object Browser
history of 2nd
Identity values, managing
Multiple Active Result Sets (MARS)
namespaces
Imports statement
Microsoft.SQL Server.Server
System.Data 2nd
System.Data.SqlClient
System.Data.SqlTypes
object instantiation
constructors
overview of
scope
Using block
overview of
populating data structures
DataSet Load method
DataTable Load method
design
hard-coding DataTable objects
InfoMessage messages
OUTPUT and RETURN value parameters
overview of
resultset elements 2nd
SqlDataAdapter Fill method
server-side cursors
creating
disadvantages of
dynamic cursors
fetching data from
fetching entire cursor
forward-only cursors
implementing
importance of
keyset cursors
managing
opening, closing, and reopening connections
static cursors
updating
SqlBulkCopy class
SqlClient performance counters
SQLClientFactory class
SqlCommand class [See SqlCommand objects.]
SqlConnection class
SqlConnectionStringBuilder class
SqlDataAdapter class
Fill method
properties
Update method
updates
SqlDataReader class
SqlDependency class
SqlError class
SqlErrorCollection class
SqlException class
SqlParameter class
SqlParameterCollection class
SqlTransaction class
TableAdapter objects
Advanced Properties dialog
Advanced Services options (SQL Server)
aggregates 2nd
CLR SUMUSD example
definition of
testing
aliases
ConnectionString
creating 2nd
AllowDelete property (DataView class)
AllowEdit property (DataView class)
AllowNew property (DataView class)
ALTER PROCEDURE statement
ampersand (&)
ANSI
compared to Unicode
data types
ANSI_NULLS setting
Append method
AppendFormat method
Appleman, Dan
application architectures [See architectures, choosing.]
application design [See also application architectures; architectures, choosing; DBMSs (database management systems), overview of.]
application settings
client/server applications
connection pools
definition of
CLR executables
concurrency
constraints
data access architectures
ASP browser-based architectures
database-agnostic applications
multi-tiered design
overview of
stand-alone applications 2nd
stand-alone applications (client/server architectures)
XMLWeb Service architectures
data restoration
backup and deployment issues
importance of
security
Data Sources, importing
development costs
example applications
aliases, creating
chapter-specific configuration issues
installing
goals of
human element
multi-user issues
overview of
population
project specifications
relational databases
resource management
scalability
security
SqlDataReader
UpdateEvents application 2nd
batch mode updates
CustomerUpdate stored procedure
DataRow versions
RowUpdated event, trapping
RowUpdating event, trapping
Update method
web applications
Application Name property (SQL Profiler)
ApplyDefaultSort property (DataView class)
approximate numbers
architectures
ADO [See ADO, ADTG Recordset, accessing.]
ADO.NET [See ADO.NET, class names.]
choosing
DAO (Data Access Objects)
data access architectures
ASP browser-based architectures
database-agnostic applications
multi-tiered design
overview of
stand-alone (client/server) applications
XMLWeb Service architectures
JET
ODBC (Open Database Connectivity)
OLEDB
Reporting Services
scalability
ASP browser-based architectures 2nd
assemblies, managing with Server Explorer
AsyncCallback objects
asynchronous execution of SqlCommand objects
BeginExecuteNonQuery function
BeginExecuteReader function
BeginExecuteScalar function
BeginExecuteXmlReader function
callback 2nd
cautions
IAsyncResult interface
overview of
polling
threads
WaitHandle objects 2nd
AsyncState property (IAsyncResult interface)
AsyncWaitHandle property (IAsyncResult interface)
AttachDBFilename keyword
attaching
example databases
.MDFdatabase files
authentication (SQL Server) 2nd
authorization (SQL Server)
Auto Close option


B

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

Back event
background color of reports
backing up databases 2nd
barbinary data type
batch mode updates
batches
BeginExecuteNonQuery function
BeginExecuteReader function
BeginExecuteScalar function
BeginTransaction method
beta software (Visual Studio)
BETWEEN expressions
BIDS (Business Intelligence Development Studio) Report Designer
bigint data type
binary data type
binding data [See data binding.]
BindingComplete event
BindingNavigator controls
BindingSource class 2nd
events
methods
CancelEdit
EndEdit
Find
MoveFirst
MoveLast
MoveNext
MovePrevious
RemoveFilter
RemoveSort
ResetBindings
ResetCurrentItem
ResetItem
ResumeBinding
properties
Current
DataMember
DataSource
Filter
Position 2nd
Sort 2nd
bit data type
BLOBs 2nd
BookmarkNavigation event
bound controls
customizing generated UI
SmartTags and
brackets ([])
breakpoints
multiple breakpoints
setting
buffer cache
BuildAndExecuteAsyncCommands routine
Business Intelligence Development Studio (BIDS) Report Designer
business rules
extended properties
overview of
strongly typed data structures and
T-SQL Rules
binding to columns
creating


C

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

cache
cached Query plans
custom report cache tables, building
CAL (Client Access License)
callback 2nd
Cancel method
CancelEdit method
canceling operations
capturing parameters
prompting for parameter values
prompting with pick lists
Cartesian products
CASCADE constraint
case-sensitive queries
Caution method
ChangeDatabase method
ChangePassword method
char data type 2nd
CHECK constraint 2nd
child/parent relationships
choosing
application architectures
connection scope
connection strategy
ConnectionStrings
data access interfaces
data access providers
SQLEv (SQL Server Everywhere Edition)
data access providers
DBMSs (database management systems)
client considerations
maintenance and administration
multi-user issues
overview of
performance
scalability
server considerations
SQL Server costs
SQL Server security
transactions-per-second limits
Query methods
class diagrams
classes
BindingNavigator
BindingSource
events
methods
properties
typical data binding scenarios
class diagrams
DataGridView
DataView
LocalReport
events
methods
properties
names
ProgressBar
Report Viewer
configuring
events
methods
overview of
properties
ServerReport
accessing on Reporting Services
methods and functions
parameters 2nd
rendering
SqlBulkCopy
SQLClientFactory
SqlCommand
SqlConnection
SqlConnectionStringBuilder
SqlContext
SqlDataAdapter
Fill method
properties
Update method
updates
SqlDataReader
SqlDependency
SqlError
SqlErrorCollection
SqlException
SqlParameter 2nd
access rights and
best practices
default parameter values
duplicate parameter names
dynamic SQL
marking parameters
NULL values
Parameters collections
ResetDbType method
ResetSqlType method
Unicode literals
validation
Value property
what can be parameterized
SqlParameterCollection 2nd
SqlTransaction
clauses [See also statements, ALTER PROCEDURE.]
COLLATE
ORDERBY
WHERE
Clear method
ClearAllPools method
ClearPool method
click-through reports
Client Access License (CAL)
client statistics
client/server architectures
advantages/disadvantages
building
connection management
connection pools
data binding
data entry paradigms
data retrieval
data storage limits
data validation
definition of
hybrid architectures
server-side cursors
stored procedures
temporary tables
clients
client statistics
client/server architectures
advantages/disadvantages
building
connection management
connection pools
data binding
data entry paradigms
data retrieval
data storage limits
data validation
definition of
hybrid architectures
server-side cursors
stored procedures
temporary tables
competing demands on
definition of 2nd
SNC (SQL Server Native Client)
SqlClient
Clone function
Close method 2nd 3rd
CloseConnection CommandBehavior (ExecuteReader function)
closing connections 2nd
automatically
connection pool connections 2nd
CLR executables
ADO.NET
Querying data
returning data/messages
SqlContext connection
transactions
aggregates 2nd
CLR SUMUSD example
definition of
testing
building
debugging
dependencies
deploying
executable dependencies, managing
functions
profiling deployment process
designing
enabling
evaluating performance of
overview of
with SQL Profiler
TVFs (table-valued functions)
examples
functions
addressing in T-SQL
deploying
TVFs (table-valued functions)
overview of 2nd
performance issues
security
stored procedures
compared to functions
CountWords example
creating
performance
returning resultsets with SqlContext class
testing
suitability for Visual Basic .NET development
suitability for Visual Studio development
test scripts
creating
executing
triggers 2nd
creating
INSERTED and DELETED pseudo tables
updates
UDTs (user-defined types)
AddLoan
challenges of
debugging
instantiating UDT variables
IsNull property
methods
Null property
NULL values
Parse function
performance tuning
public properties 2nd
public structure declaration
referencing
referencing and updating values
SELECT queries
SqlUserDefinedTypeAttribute
stepping through code
templates
ToString function
typICurrencyV2 UDT
user-defined serialization
validating data
when to use
viewing
with Server Explorer
with SQL Server Management Studio Object Explorer
when to use
COLLATE clause
collating sequences (SQL Server)
collections
DataBindings
Parameters
color of reports
columns
aliases
binding rules to
constraints 2nd
definition of
hiding
identifiers
Identity columns
concurrency management
definition of
naming
order of, changing
SqlDataReader columns, fetching
strongly typed data columns
CommandBehaviors (ExecuteReader function)
CloseConnection
Default
KeyInfo
SchemaOnly
SequentialAccess
SingleResult
SingleRow
CommandBuilder
action commands, generating
adding to applications
capabilities of
concurrency management
example of
wizard dependency on
commands [See methods and functions, Accumulate; statements, ALTER PROCEDURE.]
CommandText property (SqlCommand object)
CommandTimeout property (SqlCommand object) 2nd
CommandType property (SqlCommand object)
comments
common instances (SQL Server Express Edition)
Common Language Runtime (CLR) executables [See CLR executables, ADO.NET.]
community tech preview (CTP) releases
CompareAllSearchableValues setting (ConflictOption property)
CompareRowVersion setting (ConflictOption property)
comparing NULL values
CompletedSynchronously property (IAsyncResult interface)
complex control binding
complex data binding
concatenated T-SQL expressions
concatenating strings
concatenation operator (&)
concurrency management 2nd
CommandBuilder
design issues
Identity column values
@@Identity global variable
ADO.NET
parent/child relationship updates
retrieving new Identity values
SQL Server
overview of
Timestamp columns
Configuration Manager (SQL Server) 2nd
configuring
connection pool keywords
Report Viewer control
server connections
for Internet access
static versus dynamic ports
Surface Area Configuration (SAC) tool
SQL Profiler
TableAdapters
Visual Studio
custom project templates
online or local help
Options dialog
saving and loading custom configurations
Server Explorer
ConflictOption property (CommandBuilder)
connection pools 2nd
activating/tuning/disabling
for client/server applications
closing connections
configuring keywords
connection timeouts
just-in-time (JIT) connections
monitoring
options
overflow
for web applications
Connection Properties dialog
Connection property
SqlCommand object 2nd
SqlDataAdapter class
Connection Timeout keyword
connections
client/server connection management
closing 2nd 3rd
connection pools 2nd
activating/tuning/disabling
for client/server applications
closing connections
configuring keywords
connection timeouts
just-in-time (JIT) connections
monitoring
options
overflow
for web applications
ConnectionStrings
building for data access providers
building with SqlConnectionStringBuilder class
building with Visual Studio
connection objects
connection pool keywords
definition of
keywords and aliases 2nd
persisting
security
server name keyword
establishing 2nd
events
exception handling
SqlException class
types of exceptions
just-in-time (JIT) strategies
limitations
MARS
methods
opening 2nd
persistent connections
properties
scope
security 2nd
server configuration
for Internet access
static versus dynamic ports
Surface Area Configuration (SAC) tool
SQL Server connections
connection pooling
connection strategies
data access interface role
debugging
MARS (Multiple Active Resultsets)
opening
overview of
protocols
server state, managing
SqlConnection objects
SqlContext
timeouts
ConnectionStrings
building for data access providers
building with SqlConnectionStringBuilder class
building with Visual Studio
connection objects
connection pool keywords
definition of
keywords and aliases 2nd
persisting
security
server name keyword
constraints 2nd 3rd
constructors
New (SqlCommand object)
consuming Web Services
ContinueUpdateOnError property (SqlDataAdapter class)
controls [See also classes, BindingNavigator; Toolbox (Visual Studio).]
BindingNavigator
BindingSource
events
methods
properties
typical data binding scenarios
DataGridView
DataSet
deprecated data access controls
finding
overview of
ProgressBar
Report Viewer
configuring
events
methods
overview of
properties
shareable and reusable components
TableAdapter
generated parameters, managing
parameters, configuring with Visual Studio
stored procedures
updating tables with
converting RDL to RDLC
ConvertToDollars function
counters [See performance counters, activating.]
CountWords stored procedure
CountWordsInString function
CREATE PROCEDURE statement
CREATE RULE statement 2nd
CreateParameter function 2nd
criteria pane (Query Designer)
Crystal Report Services
CTP (community tech preview) releases
Current property (BindingSource class)
CurrentChanged event
CurrentItemChanged event
CurrentPage property (Report Viewer class)
cursor data type
cursors
cursor data type
server-side cursors
ADO.NET implementation of
creating
disadvantages of
dynamic cursors
fetching data from
fetching entire cursor
forward-only cursors
importance of
keyset cursors
managing
opening, closing, and reopening connections
static cursors
updating
custom controls, binding to
custom report cache tables, building
custom Visual Studio configurations
creating with Options dialog
custom project templates
online or local help
saving and loading
CustomerByState stored procedure 2nd
CustomerUpdate stored procedure
CustomerUpdate1 stored procedure


D

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

D&D (drag-and-drop) data binding 2nd
DAAB (Data Access Application Block)
DACW (DataAdapter Configuration Wizard)
dependency on CommandBuilder
installing
DAO (Data Access Objects)
history of
performance
Data Access Application Block (DAAB)
data access architectures
ASP browser-based architectures
DAAB (Data Access Application Block)
data access providers
database-agnostic applications
deprecated data access controls
multi-tiered designs
overview of
stand-alone (client/server) applications
advantages/disadvantages
client/server connection management
client/server data binding 2nd
client/server data entry paradigms
client/server data retrieval
client/server data validation
data storage limits
hybrid architectures
server-side cursors
stored procedures
temporary tables
XMLWeb Service architectures
data access interfaces
choosing
creating
adding parameters to queries
configuring TableAdapters
selecting Query methods
single-table TableAdapters
stored procedures
data access providers
building ConnectionStrings for
choosing 2nd
need for
SNC (SQL Server Native Client)
SQLEv (SQL Server Everywhere Edition)
implementation
overview of
programming and maintenance
security
sharing and replication
SQL Server tools
SQLEngine
Visual Studio support
Data Access Objects (DAO)
history of
performance
data access providers
building ConnectionStrings for
choosing 2nd
data binding
BindingNavigator control
BindingSource class
events
methods
properties
typical data binding scenarios
complex data binding
control properties
to custom controls
DataBindings collection
DataGridView control
definition of
drag-and-drop (D&D) data binding
example of
IBindableComponent interface
overview of
ProgressBar control
rules to columns
simple data binding
to TableAdapter
with drag and drop
without drag and drop
Data Binding with Windows Forms 2.0: Programming Smart Client Data Applications with .NET
Data Connections [See connections, client/server connection management.]
data restoration [See restoring data, backup and deployment issues.]
data retrieval
Data Source Configuration Wizard 2nd 3rd
consuming and testing Web Services
single-table TableAdapters, building
TableAdapters
adding parameters to queries
configuring
selecting Query methods
stored procedures
Data Source XSD designer 2nd 3rd
Data Sources [See also TableAdapter objects.]
creating 2nd
adding parameters to queries
configuring TableAdapters
selecting Query methods
single-table TableAdapters
stored procedures
definition of
importing
overview of
SqlClient
vulnerabilities of
Web Services
building
consuming and testing
exposing in the UI
data structures, strongly typed [See also TableAdapter objects.]
building Web Services
consuming and testing Web Services
exposing Web Services Data Sources in the UI
need for
data types
ANSI 2nd
barbinary
bigint
binary
bit
char 2nd
CLR UDTs (user-defined types)
AddLoan
challenges of
debugging
instantiating UDT variables
IsNull property
methods
Null property
NULL values
Parse function
performance tuning
public properties 2nd
public structure declaration
referencing
referencing and updating values
SELECT queries
SqlUserDefinedTypeAttribute
stepping through code
templates
ToString function
typICurrencyV2 UDT
user-defined serialization
validating data
when to use
cursor
datetime
decimal
doubleprecision
exact numerics
float
image
imprecise numbers
int
money
nchar
ntext
numeric
nvarchar
real
smalldatetime
smallint
smallmoney
sql_variant 2nd
sysname
table
text
timestamp
tinyint
UDTs (user-defined types) 2nd
Unicode 2nd
uniqueidentifier
varchar 2nd
xml
data validation
DataAccess setting (SqlMethodAttribute property)
DataAdapter Configuration Wizard [See DACW (DataAdapter Configuration Wizard), dependency on CommandBuilder.]
database connections [See connections, client/server connection management.]
Database Diagrams
creating
definition of
Database Snapshots
database-agnostic applications
databases [See relational databases, administrative functions, Auto Close option.]
DataBindings collection
DataColumn objects
DataError event
DataGridView controls
DataMember property (BindingSource class)
DataMemberChanged event
DataRow objects 2nd
DataSet controls 2nd [See also populating data structures, DataSet Load method.]
Load method
overview of 2nd
populating
typed DataSets 2nd
untyped DataSets
DataSets block (RDLC files)
DataSource property 2nd
DataSourceChanged event
DataSources property (LocalReport class)
DataTable objects
DataColumns
DataRows
joins
Load method
overview of 2nd
populating [See populating data structures, DataSet Load method.]
populating DataSets with
properties
DataTableMappingCollection property (SqlDataAdapter class)
DataView class
DataWorks team
dates
data types
inqueries
datetime data type
db_datareader role
db_datawriter role
db_owner role
db_public role
DBCC
DBMSs (database management systems)
client considerations
maintenance and administration
multi-user issues
overview of
performance
.NET Framework
acceptable speed
ADO
DAO
developer performance
JET
MDAC stack, sharing
overview of
performance metrics
shareable and reusable components
shared code
SQL Server
team development
scalability
server considerations
SQL Server costs
SQL Server security
transactions-per-second limits
DDL triggers
debugging
CLR executables
SQL Server connections
stored procedures
against local instances
with breakpoints
from code
on remote instances
UDTs (user-defined types)
decimal data type
DECLARE statement
Default CommandBehavior (ExecuteReader function)
default instances (SQL Server Express Edition)
default parameter values
DEFAULT values (INSERT statement)
DefaultValue property (DataColumn objects)
Delaney, Karen
Delegate objects
Delete Query type (Query Designer)
DELETE statements
DeleteCommand object
DeleteCommand property (SqlDataAdapter class)
DELETED pseudo table
denying permissions
DenyPermissionsOnTo stored procedure
dependencies (CLR executables)
deployment
CLR executables
executable dependencies
functions
profiling deployment process
data restoration
SQL Server Express Edition
deprecated controls
reinstalling
derived tables
design [See application design, application settings.]
destroying connection pools
Developer Edition (SQL Server)
developers
keeping skills up to date
optimizing performance of
diagram pane (Query Designer)
diagrams
class diagrams
Database Diagrams
creating
definition of
dialogs [See specific dialogs.]
disabling connection pools
DisplayMember property
DisplayName property 2nd
Dispose method 2nd 3rd 4th
Disposed event (connections)
distribution database
DML triggers
documentation
MSDN documentation
weaknesses of
DocumentMapCollapsed property (Report Viewer class)
DocumentmapNavigation event
DocumentMapWidt property (Report Viewer class)
domains
doubleprecision data type
DoWork event handler
drag-and-drop (D&D) data binding 2nd
drill-through reports
Drillthrough event handler 2nd
DSCW [See Data Source Configuration Wizard.]
duplicate parameter names
duplicating SqlCommand objects
DVD contents
aliases, creating
chapter-specific configuration issues
example applications, installing
example databases
attaching
installing
dynamic cursors
dynamic ports
dynamic SQL


E

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

EBooks
editing tables with Server Explorer
EDM (Entity Data Modeling)
embedded spaces, avoiding in object names
EnableExternalImages property (LocalReport class)
EnableHyperlinks property (LocalReport class)
enabling
CLR executables
connection pools
User Instances
End User License Agreement (EULA)
EndEdit method
Enterprise Edition (SQL Server)
Entity Data Modeling (EDM)
error handling [See exception handling, connection pool overflows.]
ESPs (Extended Stored Procedures)
EULA (End User License Agreement)
evaluating performance [See performance, CLR executable performance, evaluating.]
event logs
events
BindingSource class
connections
DoWork
Drillthrough
ListChanged
LocalReport events
Report Viewer events
RowUpdated
RowUpdating
executing Update logic with
trapping
Events Selection dialog (SQL Profiler)
exact numerics
Exception class
exception handling
connection pool overflows
connections
SqlException class
types of exceptions
executables [See CLR executables, ADO.NET.]
ExecuteNonQuery function
ExecuteReader function 2nd
CommandBehavior.CloseConnection
CommandBehavior.Default
CommandBehavior.KeyInfo
CommandBehavior.SchemaOnly
CommandBehavior.SequentialAccess
CommandBehavior.SingleResult
CommandBehavior.SingleRow
ExecuteReportInCurrentAppDomain method
ExecuteReportInSandboxAppDomain method
ExecuteScalar function 2nd
ExecuteUpdateEvents BatchMode example
ExecuteXmlReader function 2nd
execution
asynchronous execution (SqlCommand objects)
BeginExecuteNonQuery function
BeginExecuteReader function
BeginExecuteScalar function
BeginExecuteXmlReader function
callback 2nd
cautions
IAsyncResult interface
overview of
polling
threads
WaitHandle objects 2nd
Query plans
SELECT queries
stored procedures 2nd
synchronous execution (SqlCommand objects) [See also populating data structures, DataSet Load method.]
connection management
ExecuteNonQuery function
ExecuteReader function 2nd
ExecuteScalar function 2nd
ExecuteXmlReader function 2nd
overview of
SqlDataReader object
test scripts
Export Template command (File menu)
Export Template Wizard
exporting data
exposing Web Services Data Sources
Express Edition (SQL Server)
administration 2nd
capabilities
common instances
configuring as server
crippled features
database deployment
default instances
deployment 2nd
independent instances
overview of 2nd
performance 2nd
replication
scalability 2nd
security 2nd
Service Packs 2nd
SystemAdministrator's password
tools 2nd
user instances
Express Edition (Visual Studio)
ExpressAdvanced Services Edition (SQL Server)
Expression Editor
Expression property (DataColumn objects)
expressions
BETWEEN
concatenated T-SQL expressions
creating for reports
IN
LIKE
extended properties (business rules)
Extended Relational Analysis
Extended Stored Procedures (ESPs)
extensibility
EXTERNAL_ACCESS setting (CLR executables)


F

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

Failover Partner keyword
FETCH operator
fetching
cursor data
cursors
resultsets
SqlDataReader columns
SqlDataReader rows
fields [See columns, aliases.]
File menu commands
Export Template
New Project
files
.MDFdatabase files, attaching
RDLC extension files
converting RDL to
DataSets block
Fill method 2nd 3rd
FillLoadOption property (SqlDataAdapter class)
FillRowMethodName setting (SqlMethodAttribute property)
Filter property (BindingSource class)
Find method 2nd
finding
controls
servers/instances to establish connections
FindRows method
firewalls, configuring
for Internet access
static versus dynamic ports
Surface Area Configuration (SAC) tool
flat files, accessing
float data type
floating point numbers 2nd
FOREIGN KEY constraint
foreign keys
Format method
formatting reports
background color
expressions
forms, binding TableAdapters
with drag and drop
without drag and drop
forward-only cursors
functions [See methods and functions, Accumulate.]


G

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

generated parameters, managing
generated UI, customizing
GetBoolean function
GetByte function
GetBytes function
GetChars function
GetData method
GetDataSourceNames function
GetDataSources function
GetDataTypeName function
GetDateTime function
GetDecimal function
GetDefaultPageSettings function 2nd
GetDocumentMap function
GetDouble function
GetFieldTypeName function
GetFloat function
GetGuid function
GetHashCode function
GetInt16 function
GetInt32 function
GetInt64 function
GetName function
GetOrdinal function
GetParameters function 2nd
GetProviderSpecificFieldType function
GetProviderSpecificValue function
GetProviderSpecificValues function
GetSchema function
GetSchemaTable function 2nd
GetServerVersion function
GetString function
GetTotalPages function 2nd
GetValue function
GetValues function
global connections
global variables, @@IDENTITY 2nd
globally unique identifiers (GUIDs)
grouping report data
GUI DataSet designer
GUIDs (globally unique identifiers)


H

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

hackers
HardConnectsPerSecond performance counter
HardDisconnectsPerSecond performance counter
HasRows function
help and support 2nd
hiding columns
history
of ADO
of ADO.NET 2nd
of DAO
of JET
of ODBC
of OLEDB
History of Basic
HistoryID parameter (server reports)
Hitchhiker's Guide to SQL Server 2000 Reporting Services
hitchhikerguides.net website
Hollis, Bill
hybrid architectures
Hyperlink event
HyperlinkTarget property (Report Viewer class)


I

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

IAsyncResult interface
IBindableComponent interface
IDENT_CURRENT() function
identifiers
GUIDs (globally unique identifiers)
Identity columns
concurrency management
@@Identity global variable 2nd
ADO.NET
parent/child relationship updates
retrieving new Identity values
SQL Server
definition of
Identity values (INSERT statement)
image data type
image library (Visual Studio), installing
Import and Export Settings command (Tools menu)
Import and Export Settings Wizard
importing data 2nd
Imports statement
imprecise numbers
IN expressions
indexes
creating
indexed views
InfoMessage event (connections)
InfoMessage messages, populating
Init function
Initial Catalog keyword
injection attacks
Inline Functions
inner joins
Insert Results Query type (Query Designer)
INSERT statement
generating with Query Designer
Identity values
NULL and DEFAULT values
Insert Values Query type (Query Designer)
InsertCommand object
InsertCommand property (SqlDataAdapter class)
INSERTED pseudo table
installing
DACW (DataAdapter Configuration Wizard)
deprecated data components
example applications
example databases
SQL Server
Advanced Services options
authentication mode
collating sequences
instance names
quiet installs
Service Account
Setup program
User Instances, enabling
SQL Server Express Edition
common instances
default instances
independent instances
Visual Studio
image library
MSDN documentation
Setup program
instances
establishing aliases to
finding for establishing connections
multiple instances
object instantiation
constructors
overview of
scope
Using block
overview of
SqlCommand instances
UDT variables
int data type
Integrated Security keyword
interfaces
IAsyncResult
IBindableComponent
Internet access
connection strategy
server/firewall configuration
intertable relationships, managing in Data Source designer
intranet access connection strategy
InvokeIfReceiverIsNull setting (SqlMethodAttribute property)
IS NOT NULL statement
IS NULL statement
IsCompleted property (IAsyncResult interface)
IsDBNull function
IsDeterministic setting (SqlMethodAttribute property)
IsDrillthroughReport property 2nd
IsMutator setting (SqlMethodAttribute property)
IsNull property (CLR UDTs)
IsPrecise setting (SqlMethodAttribute property)


J

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

JET
database access
history of
performance
JIT (just-in-time) connection strategies
JOIN operator
joins
Cartesian products
DataTable objects
inner joins
outer joins
SELECT queries
JumpToBookmark method
JumpToDocumentMapID method
just-in-time (JIT) connection strategies


K

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

KeyInfo CommandBehavior (ExecuteReader function)
keys, primary
changing
defining
GUIDs (globally unique identifiers)
multi-column primary keys
primary key/foreign key relationships
keyset cursors
keywords [See also statements, ALTER PROCEDURE.]
connection pools
ConnectionString
configuring connection pool keywords
list of 2nd
server name keyword


L

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

launching [See starting, SQL Profiler traces.]
laying out reports 2nd
legal issues, Privacy Act of 1974
libraries, Visual Studio image library
LIKE expressions
limiting user choices
ListChanged event 2nd
ListRenderingExtensions method 2nd
lists, pick lists
literals
inqueries
Unicode literals
Load method
loading Visual Studio custom configurations
LoadReportDefinition method 2nd
LoadSubreportDefinition method
local access connection strategy
local connections
local instances, T-SQL debugging
LocalReport class
events
methods
properties
LocalReport property (Report Viewer class)
logins (SQL Server)
logs
event logs
transaction logs
LOWER function


M

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

maintenance
DBMSs
in SQLEv (SQL Server Everywhere Edition)
Make Table Query type (Query Designer)
Management Classes
Management Studio (SQL Server)
creating queries
Object Explorer
queries [See queries, ad hoc queries, integrating into applications.]
stored procedure management
managing
concurrency
CommandBuilder
Identity column values
overview of
generating parameters
parameters
resources
server report parameters 2nd
server-side cursors 2nd
servers with Server Explorer
SqlConnection objects
stored procedures
temporary tables
user databases
market share
marking parameters
MARS (Multiple Active Resultsets) 2nd 3rd 4th
master databases
Matric report
building
custom report cache table
data considerations
grouping data
initial SELECT Query
layout
testing 2nd
maximum connection pool size
MaxLength property (DataColumn objects)
MDAC stack, sharing
.MDFdatabase files, attaching
memory
buffer cache
Shared Memory
Merge function
Message Queuing
messages
InfoMessage messages, populating
Message Queuing
metadata, fetching from SqlDataReader streams
methods and functions 2nd [See also stored procedures.]
Accumulate
Add 2nd
AddRange 2nd
AddTrustedCodeModuleInCurrentAppDomain
Append
AppendFormat
BeginExecuteNonQuery
BeginExecuteReader
BeginExecuteScalar
BeginExecuteXmlReader
BuildAndExecuteAsyncCommands
Cancel
CancelEdit
Clear
Clone
Close
CLR aggregates
CLR functions
addressing in T-SQL
deploying
UDT methods
compared to stored procedures
connections
ConvertToDollars
CountWordsInString
CreateParameter 2nd
Dispose 2nd
EndEdit
ExecuteNonQuery
ExecuteReader 2nd
CommandBehavior.CloseConnection
CommandBehavior.Default
CommandBehavior.KeyInfo
CommandBehavior.SchemaOnly
CommandBehavior.SequentialAccess
CommandBehavior.SingleResult
CommandBehavior.SingleRow
ExecuteReportInCurrentAppDomain
ExecuteReportInSandboxAppDomain
ExecuteScalar 2nd
ExecuteXmlReader 2nd
Fill 2nd
Find 2nd
FindRows
Format
GetBoolean
GetByte
GetBytes
GetChars
GetDataSourceNames
GetDataSources
GetDataTypeName
GetDateTime
GetDecimal
GetDefaultPageSettings 2nd
GetDocumentMap
GetDouble
GetFieldTypeName
GetFloat
GetGuid
GetHashCode
GetInt16
GetInt32
GetInt64
GetName
GetOrdinal
GetParameters 2nd
GetProviderSpecificFieldType
GetProviderSpecificValue
GetProviderSpecificValues
GetSchemaTable 2nd
GetServerVersion
GetString
GetTotalPages 2nd
GetValue
GetValues
HasRows 2nd
IDENT_CURRENT()
Init
instructors
IsDBNull
JumpToBookmark
JumpToDocumentMapID
ListRenderingExtensions 2nd
Load
LoadReportDefinition 2nd
LoadSubreportDefinition
LOWER
Merge
MoveFirst
MoveLast
MoveNext
MovePrevious
MovePrevoius
New
NewRow
Parse 2nd
PerformBack
Prepare 2nd
Read
Refresh 2nd
RefreshReport
RemoveFilter
RemoveSort
Render 2nd
RenderStream
Reset
ResetBindings
ResetCommandTimeout 2nd
ResetCurrentIrem
ResetDbType
ResetItem
ResetSqlType
ResumeBinding
Set Parameters
SetDataSourceCredentials
SetParameters
Sleep
T-SQL Functions
advantages of
Inline Functions
return values
Scalar Functions
structure of
Table Functions
Terminate
ToString 2nd
ToTable
TVFs (table-valued functions)
Update 2nd
UPPER
WaitAll
WaitAny
metrics (performance)
Microsoft
Learning team
product documentation
product updates
Microsoft.SQL Server.Server namespace
minimum connection pool size
MissingMapping Action property (SqlDataAdapter class)
MissingSchemeAction property (SqlDataAdapter class)
Mixed Mode (SQL Server)
model database
money data type
monitoring SQL Server
connection pools
overview of
with performance counters
activating performance counters
selecting counters
SqlClient performance counters
with SQL Profiler
Application Name property
configuring
evaluating traces
filter criteria, adding
starting traces
tips and recommendations
MoveFirst method
MoveLast method
MoveNext method
MovePrevious method
msdb database
MSDE Edition (SQL Server) [See Express Edition (SQL Server), administration.]
MSDN documentation, installing
MSDN Search Index Utility
multi-column primary keys
multi-tiered application design
multi-user issues
Multiple Active Resultsets (MARS) 2nd 3rd 4th
multiple breakpoints, setting
multiple databases
multiple SQL Server instances


N

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

n-tiered application design
Name setting (SqlMethodAttribute property)
Named Pipes
names
aliases
class names
object names
stored procedures
System.Data namespace naming conventions
namespaces
class names
Imports statement
Microsoft.SQL Server.Server
System.Data
DataSet objects 2nd
DataTable objects 2nd
DataView class
joins
naming conventions
overview of
System.Data.SqlClient
SqlBulkCopy class
SQLClientFactory class
SqlCommand class
SqlConnection class
SqlConnectionStringBuilder class
SqlDataAdapter class
SqlDataReader class
SqlDependency class
SqlError class
SqlErrorCollection class
SqlException class
SqlParameter class
SqlParameterCollection class
SqlTransaction class
System.Data.SqlTypes
nchar data type
.NET Framework
performance
Web Services and
New constructor
connections
SqlCommand object
New Project command (File menu)
New Project dialog
NewRow method
NIH (Not Invented Here) syndrome
NO ACTION constraint
normalization
definition of
first normal form
second normal form
third normal form
NOT NULL constraint
Notification property (SqlCommand object) 2nd
NotificationAutoEnlist property (SqlCommand object) 2nd
ntext data type
NULL constraint
Null property (CLR UDTs)
NULL values 2nd
CLR UDT (user-defined type) values
INSERT statement
parameter-based queries
NumberOfActiveConnectionPoolGroups performance counter
NumberOfActiveConnectionPools performance counter
NumberOfActiveConnections performance counter
NumberOfFreeConnections performance counter
NumberOfInactiveConnectionPoolGroups performance counter
NumberOfInactiveConnectionPools performance counter
NumberOfNonPooledConnections performance counter
NumberOfPooledConnections performance counter
NumberOfStasisConnections performance counter
numbers
approximate numbers
dates
decimals 2nd
exact numerics
floating point numbers 2nd
imprecise numbers
money
numeric data type
nvarchar data type


O

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

Object Browser
class diagrams, creating
Object Explorer
CLR executables, viewing
compared to Visual Studio Server Explorer
objects [See also classes, BindingNavigator; controls, BindingNavigator.]
addressing in SQL Server
AsyncCallback
DataColumn
DataRow 2nd
DataSet
Load method
overview of 2nd
populating
strongly typed DataSets
DataTable
DataColumns
DataRows
joins
Load method
overview of 2nd
populating DataSets with
populating from SqlDataReader
Delegate
DeleteCommand
disposing of
duplicating
InsertCommand
instantiating
constructors
overview of
scope
Using block
ownership
populating [See populating data structures, DataSet Load method.]
SelectCommand
SqlCommand [See SqlCommand objects.]
SqlConnection
SqlDataReader
best practice designs
fetching all columns
fetching individual rows
fetching metadata
Get functions
GetSchemaTable method
HasRows function
overview of
testing stream for data 2nd
TableAdapter 2nd 3rd
adding parameters to queries
binding to
configuring
defined
generated parameters, managing
intertable relationships, managing in Data Source designer
limitations of
overview of 2nd
parameters, configuring with Visual Studio
selecting Query methods
single-table TableAdapters
stored procedures 2nd 3rd
updating tables with
UpdateCommand
WaitHandle 2nd
ODBC (Open Database Connectivity)
OLEDB
OnNullCall setting (SqlMethodAttribute property)
Open Database Connectivity (ODBC)
Open method
Open Table Definition dialog
opening connections
automatically
SQL Server connections 2nd
operations, canceling
operators
BETWEEN
FETCH
IN
JOIN
LIKE
TRUNCATE TABLE
Using
optimizing performance [See performance, CLR executable performance, evaluating.]
Options command (Tools menu)
Options dialog (Visual Studio)
ORDER BY clause
order of columns, changing
OriginalParametersToDrillthrough property (LocalReport class)
outer joins
OUTPUT parameters
overflow (connection pools)
OverwriteChanges setting (ConflictOption property)
ownership 2nd
Ozzie, Ray


P

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

PageNavigation event
parameter-based queries
access rights and
ad hoc queries, inserting
best practices
default parameter values
duplicate parameter names
dynamic SQL
marking parameters
NULL values
parameter-based updates
Parameters collections
query parameters
adding
capturing
definition of
SELECT queries
SqlParameter class
SqlParameterCollection class
Unicode literals
validation
overview of
sample application
Value property
what can be parameterized
parameters
queries [See parameter-based queries.]
report parameters
creating
definition of
sample application
Parameters collections
Parameters property (SqlCommand object) 2nd
parent/child relationships
updating
Parse function 2nd
Password keyword
passwords, System Administrators
patches
percent sign (%)
performance
CLR executable performance, evaluating
overview of
with SQL Profiler
stored procedures
TVFs (table-valued functions)
UDTs (user-defined types)
connection pools
DBMSs
.NET Framework
acceptable speed
ADO
DAO
developer performance
JET
MDAC stack, sharing
overview of
performance metrics
shareable and reusable components
shared code
team development
performance counters
activating
selecting 2nd
SqlClient performance counters
viewing with Server Explorer
Query plans
SQL Server Express Edition 2nd
performance counters
activating
selecting
SqlClient performance counters
viewing with Server Explorer
PerformBack method
permissions, denying
Persist Security Info keyword
persistent connections 2nd
personas (security)
pick lists, prompting with
PID (process identifier)
plans, Query [See Query plans.]
polling
pools [See connection pools.]
populating data structures
DataSet Load method
DataSets with DataTable objects
DataTable Load method
design
hard-coding DataTable objects
InfoMessage messages
OUTPUT and RETURN value parameters
overview of
resultset elements 2nd
SqlDataAdapter Fill method
ports
Position property (BindingSource class) 2nd
PositionChanged event
Prepare method 2nd
PRIMARY KEY constraint
primary keys
changing
defining
GUIDs (globally unique identifiers)
multi-column primary keys
PRIMARY KEY constraint
primary key/foreign key relationships
Print event
Privacy Act of 1974
prjSampleVSReport application
adding to project
laying out
procedures, stored [See stored procedures.]
ProcessingMode property (Report Viewer class)
ProductByProfit report
Data Sources, creating
formatting
background color
expressions
laying out
Report Viewer control configuration
testing
Professional Edition (Visual Studio)
Profiler [See SQL Profiler.]
projects
adding reports to
specifications
PromptAreaCollapsed property (Report Viewer class)
prompting
for parameter values
with pick lists
properties [See specific properties.]
protection [See security, application design.]
protocols
Named Pipes
Shared Memory
TCP/IP (Transmission Control Protocol/Internet Protocol)
VIA (Virtual Interface Adaptor)


Q

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

QO (Query Optimizer) [See also Query plans.]
queries [See also Query Designer, criteria pane; statements, ALTER PROCEDURE.]
ad hoc queries, integrating into applications 2nd
adding parameters to
aliasing
BETWEEN expressions
case-sensitive queries
concatenated T-SQL expressions
creating
dates
IN expressions
LIKE expressions
literals
managing with Query Designer
criteria pane
Delete Query type
derived tables
derived views
diagram pane
Insert Results Query type
Insert Values Query type
JOIN behavior
Make Table Query type
overview of
Select Query type
SQL pane
Update Query type
NULL values
parameter-based queries
access rights and
ad hoc queries, inserting
best practices
default parameter values
duplicate parameter names
dynamic SQL
marking parameters
NULL values
Parameters collections
SqlParameter class
SqlParameterCollection class
Unicode literals
Value property
what can be parameterized
parameters
capturing
definition of
pick-list queries
QO (Query Optimizer)
Query plans
cached Query plans
client statistics and alternate plans
executing
tuning
viewing
report queries
resultsets, fetching
SELECT queries
CLR UDTs (user-defined types)
creating
executing
generating with Query Designer
joins 2nd
limiting number of rows returned
ORDER BY clause
ordering rows returned
parameters
WHERE clause
selecting methods
SQL injection attacks, preventing
sql_variant datatype
UNION queries
UPDATE queries
BLOBs
concurrency
decimal and floating point numbers
generating with Query Designer
parameter-based updates
SET clause
strings
strongly typed data columns
updating cursors
validation
overview of
sample application
Query Designer
criteria pane
Delete Query type
derived tables
diagram pane
Insert Results Query type
Insert Values Query type
JOIN behavior
Make Table Query type
managing database views with
overview of
Select Query type
SQL pane
Update Query type
Query Optimizer (QO)
Query plans
cached Query plans
client statistics and alternate plans
executing
tuning
viewing
Quick Basic
quiet installs (SQL Server)


R

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

RDL (report definition language)
converting to RDCL
RDLC extension files
converting RDL to
DataSets block
Read method
real data type
records affected value, capturing
Reflector
Refresh method 2nd
RefreshReport method
registering servers with Server Explorer
reinstalling
DACW (DataAdapter Configuration Wizard)
deprecated data components
relational databases [See also ADO.NET, class names; DBMSs (database management systems), client considerations.]
administrative functions
Auto Close option
importing/exporting data
.MDF database files, attaching
backing up
BLOBs, avoiding
check constraints
columns
binding rules to
definition of
strongly typed data columns
concurrency
constraints
creating in code
creating with Server Explorer
data types
ANSI 2nd
barbinary
bigint
binary
bit
char 2nd
cursor
datetime
decimal
doubleprecision
exact numerics
float
image
imprecise numbers
int
money
nchar
ntext
numeric
nvarchar
real
smalldata
smallint
smallmoney
sql_variant
sysname
table
text
timestamp
tinyint
UDTs (user-defined types)
Unicode 2nd
uniqueidentifier
varchar 2nd
xml
database connections [See connections, client/server connection management.]
Database Diagrams
creating
definition of
definition of
design principles
example databases
aliases
attaching
chapter-specific configuration issues
installing
identifiers
indexes
maximum size of
multiple databases
normalization
definition of
first normal form
second normal form
third normal form
object names
overview of
parent/child relationships
primary keys
changing
defining
GUIDs (globally unique identifiers)
multi-column primary keys
primary key/foreign key relationships
rows
definition of
dropping all rows
inserting 2nd
limiting number of rows returned
ordering rows returned
row-overflow storage
updating
rules
system databases
tables [See tables, aliases.]
transaction logs
transactions 2nd
updating [See updates.]
user databases
creating
managing
relationships, managing in Data Source designer
releasing connection pool connections
remote debugging
RemoveFilter method
RemoveSort method
Render method 2nd
rendering server reports
RenderingBegin event
RenderingComplete event
RenderStream method
replication
in SQL Server Express Edition
in SQLEv (SQL Server Everywhere Edition)
SqlCommand objects
report definition language [See RDL (report definition language).]
Report Designer (BIDS) 2nd
Report Items toolbox
report parameters
creating
definition of
sample application
Report Viewer
configuring
events
methods
overview of
properties
ReportEmbeddedResource property (LocalReport class)
ReportError event
ReportExport event
Reporting Services
architecture
BIDS Report Designer 2nd
features
overview of
RDL (report definition language)
sample summary report
ReportPath parameter (server reports)
reports
adding to projects
BIDS Report Designer 2nd
click-through reports
Data Sources
expressions
formatting
laying out
LocalReport class
events
methods
properties
Matrix report
building
custom report cache table
data considerations
grouping data
initial SELECT Query
layout
testing
overview of
Query parameters
RDL (report definition language)
converting to RDCL
RDLC extension files
converting RDL to
DataSets block
Report Items toolbox
report parameters
creating
definition of
sample application
report queries
Report Viewer
configuring
events
methods
overview of
properties
Reporting Services
architecture
features
overview of
sample summary report
server reports
accessing on Reporting Services
methods and functions
parameters 2nd
rendering
ServerReport class
SubReport item
testing
ReportServer database
ReportServerCredentials parameter (server reports)
ReportServerUrl parameter (server reports)
reserved words, avoiding in object names
Reset method
ResetBindings method
ResetCommandTimeout method 2nd
ResetCurrentItem method
ResetDbType method
ResetItem method
ResetSqlType method
resetting server report parameters
resources, managing
restoring data
backup and deployment issues
importance of
security
resultsets
CLR stored procedures
fetching
MARS (Multiple Active Resultsets)
ResumeBinding method
retrieving data
Identity values
RETURN values
populating
stored procedures
ReturnProviderSpecificTypes property (SqlDataAdapter class)
reusable components, building
Roeder, Lutz
roles (SQL Server)
RowFilter property (DataView class)
rows
definition of
dropping all rows
inserting
Identity values
NULL and DEFAULT values
SELECT INTO syntax
limiting number of rows returned
ordering rows returned
row-overflow storage
SqlDataReader rows, fetching
updating
BLOBs
concurrency
decimal and floating point numbers
parameter-based updates
strings
strongly typed data columns
UPDATE SET syntax
RowState property (DataRow objects)
RowStateFilter property (DataView class)
RowUpdated event
RowUpdating event
executing Update logic with
trapping
rules
business rules
extended properties
overview of
T-SQL Rules
binding to columns
creating
Run Selection feature
running report queries


S

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

SAC (Surface Area Configuration) tool
SAFE setting (CLR executables)
saving Visual Studio custom configurations
scalability
application design
architectures
SQL Server Express Edition
Scalar Functions 2nd
SchemaOnly Command Behavior (ExecuteReader function)
schemas
SQL Server 2nd
scope
for connections
selecting
object instantiation
scripts
CLRtest scripts
creating
executing
security
application design
CLR executables
connection pools
ConnectionStrings 2nd
data restoration
hackers
of connections
selecting connection strategy
Slammer worm
SQL Server
addressing objects
authentication
authorization
connections
logins 2nd
object ownership
overview of
ownership chains
roles
schemas
security personas
special accounts
usernames
SQL Server Express Edition 2nd
SQLEv (SQL Server Everywhere Edition)
SQLinjection attacks, preventing
security personas, creating
SELECT queries
CLR UDTs (user-defined types)
creating
executing
generating with Query Designer
JOIN behavior
joins 2nd
Cartesian products
inner joins
outer joins
limiting number of rows returned
ORDER BY clause
ordering rows returned
parameters
WHERE clause
SELECT statement
creating cursors with
SELECT INTO
SelectCommand object
SelectCommand property (SqlDataAdapter class)
selecting [See choosing, application architectures.]
SequentialAccess CommandBehavior (ExecuteReader function)
serialization
Server Explorer
assemblies, managing
CLR executables, viewing
compared to SQL Server Object Explorer
configuring
creating databases with
database connections
Database Diagrams
creating
definition of
servers
managing
registering
stored procedures
creating complex stored procedures
creating simple stored procedures
CustomerByState 2nd
CustomerUpdate1
debugging against local instances
debugging from code
debugging on remote instances
debugging with breakpointss
executing with Run Selection feature
naming
Synonyms
T-SQL Functions
advantages of
Inline Functions
return values
Scalar Functions
structure of
Table Functions
tables, editing
UDTs (user-defined types), creating
Server Explorer command (View menu)
server name keyword
server reports
accessing on Reporting Services
methods and functions
parameters 2nd
rendering
ServerReport class
server-side cursors
ADO.NET implementation of
creating
disadvantages of
dynamic cursors
fetching data from
fetching entire cursor
forward-only cursors
importance of
keyset cursors
managing
opening, closing, and reopening connections
static cursors
updating
server-side logic updates
executing with RowUpdating event
overview of
UpdateEvent sample application
CustomerUpdate stored procedure
DataRow versions
Update application
Update method
ServerReport class [See server reports, accessing on Reporting Services.]
ServerReport property (Report Viewer class)
servers [See also client/server architectures, advantages/disadvantages; SQL Server, action commands.]
competing demands on
definition of 2nd
finding for establishing connections
managing with Server Explorer
registering with Server Explorer
Service Accounts (SQL Server)
Service Packs 2nd
services
connecting to service instances
inspecting
starting
viewing with Server Explorer
SET DEFAULT constraint
SET NULL constraint
SetDataSourceCredentials method
SetParameters method 2nd
Setup program (SQL Server)
Setup program (Visual Studio)
sharing
in SQLEv (SQL Server Everywhere Edition)
MDAC stack
shareable components, building
shared code performance
Shared Memory
ShowCredentialPrompts property (Report Viewer class)
ShowDocumentMapButton property (Report Viewer class)
ShowExportControls property (Report Viewer class)
ShowFindControlsproperty (Report Viewer class)
ShowPageNavigationControls property (Report Viewer class)
ShowPrintButton property (Report Viewer class)
ShowPromptAreaButton property (Report Viewer class)
ShowRefreshButton property (Report Viewer class)
ShowReportBody property (Report Viewer class)
ShowToolbar property (Report Viewer class)
ShowZoomControl property (Report Viewer class)
Simple Object Access Protocol (SOAP)
SingleResult CommandBehavior (ExecuteReader function)
SingleRow CommandBehavior (ExecuteReader function) 2nd
SKUs (stock-keeping units)
Slammer worm 2nd
Sleep function
smalldatetime data type
smallint data type
smallmoney data type
Smart Tags
SNC (SQL Server Native Client)
SOAP (Simple Object Access Protocol)
SoftConnectsPerSecond performance counter
SoftDisconnectsPerSecond performance counter
Sort property
BindingSource class
DataView class
sp_addextendedproperty stored procedure
sp_dropextendedproperty stored procedure
sp_prepexec stored procedure
sp_updateextendedproperty stored procedure
special accounts (SQL Server)
specifications (project)
SQL engine (SQLEv)
SQL injection attacks
SQL pane (Query Designer)
SQL Profiler
Application Name property
CLR executable performance, evaluating
configuring
evaluating traces
filter criteria, adding
profiling CLR executable deployment process 2nd
starting traces 2nd
tips and recommendations
SQL Server
action commands
INSERT
UPDATE
administrative functions
Auto Close option
backups
importing/exporting data
.MDF database files, attaching
batches
buffer cache
business rules
extended properties
overview of
T-SQL Rules
CLR executables [See CLR executables, ADO.NET.]
comparison of editions
Configuration Manager 2nd
connections [See connections, client/server connection management.]
constraints
costs
Developer Edition
Enterprise Edition
Express Edition
administration 2nd
capabilities
common instances
configuring as server
crippled features
database deployment
default instances
deployment
eployment
independent instances
overview of 2nd
performance 2nd
replication
scalability 2nd
security 2nd
Service Packs 2nd
System Administrator's password
tools 2nd
user instances
ExpressAdvanced Services Edition
Identity values, managing
installing
Advanced Services options
authentication mode
collating sequences
instance names
quiet installs
Service Account
Setup program
User Instances, enabling
instances
multiple instances
overview of
logins
Management Studio
CLR executables, viewing
creating queries
stored procedure management
monitoring with performance counters
activating performance counters
selecting counters
SqlClient performance counters
monitoring with SQL Profiler
Application Name property
configuring
evaluating traces
evaluting CLR executable performance
filter criteria, adding
profiling CLR executable deployment process
starting traces
tips and recommendations
object addressing
Object Explorer
object ownership
overview of
ownership chains
performance
QO (Query Optimizer)
queries [See queries, ad hoc queries, integrating into applications.]
Query plans
cached Query plans
client statistics and alternate plans
executing
tuning
viewing
Reporting Services [See Reporting Services, architecture.]
roles
schemas
scripts
security
addressing objects
authentication
authorization
connections
injection attacks
logins
object ownership
overview of
ownership chains
roles
schemas
security personas
special accounts
usernames
services
connecting to service instances
inspecting
starting
special accounts
SQLEv (SQL Server Everywhere Edition)
implementation
overview of
programming and maintenance
security
sharing and replication
SQL Server tools
SQLEngine
Visual Studio support
Standard Edition
statements [See statements, ALTER PROCEDURE.]
stored procedures [See stored procedures.]
system databases
TDS (Tabular Data Stream)
transaction logs
transactions
triggers
DDL triggers
DML triggers
UDTs (user-defined datatypes)
user databases
creating
managing
usernames
views
creating
Database Snapshots
indexes
T-SQL views
Workgroup Edition 2nd
SQL Server Everywhere 2nd
SQL Server Everywhere Edition [See SQLEv (SQL Server Everywhere Edition), implementation.]
SQL Server Management Studio
SQL Server Native Client (SNC)
sql_variant data type
sql_variant datatype
SqlBulkCopy class
SqlClient namespace
ADO.NET performance counters
overview of 2nd
SqlBulkCopy class
SQLClientFactory class
SqlCommand class
SqlConnection class
SqlConnectionStringBuilder class
SqlDataAdapter class
Fill method
properties
Update method
updates
SqlDataReader class
SqlDependency class
SqlError class
SqlErrorCollection class
SqlException class
SqlParameter class
SqlParameterCollection class
SqlTransaction class
SQLClientFactory class
SqlCommand objects
ad hoc queries, integrating into applications 2nd
creating
executing asynchronously
BeginExecuteNonQuery function
BeginExecuteReader function
BeginExecuteScalar function
BeginExecuteXmlReader function
callback 2nd
cautions
IAsyncResult interface 2nd
overview of
polling
threads
WaitHandle objects 2nd
executing synchronously [See also populating data structures, DataSet Load method.]
connection management
ExecuteNonQuery function
ExecuteReader function
ExecuteScalar function 2nd
ExecuteXmlReader function 2nd
overview of
SqlDataReader object
methods
Cancel
Dispose
New
Prepare 2nd
ResetCommandTimeout 2nd
parameter-based queries
access rights and
ad hoc queries, inserting
best practices
default parameter values
duplicate parameter names
dynamic SQL
marking parameters
NULL values
Parameters collections
SqlParameter class
SqlParameterCollection class
Unicode literals
Value property
what can be parameterized
properties
CommandText
CommandTimeout 2nd
CommandType
Connection 2nd
Notification 2nd
NotificationAutoEnlist 2nd
Parameters 2nd
Transaction 2nd
UpdatedRowSource 2nd
SQL injection attacks, preventing
stored procedures
executing
RETURN values
support functions
Clone
CreateParameter 2nd
validation
overview of
sample application
SqlCommandBuilder [See CommandBuilder, action commands, generating.]
SqlConnection class [See connections, client/server connection management.]
SqlConnectionStringBuilder class 2nd
SqlContext class 2nd
SqlDataAdapter class
Fill method
properties
Update method
updates
SqlDataReader class
best practice designs
fetching all columns
fetching individual rows
fetching metadata
Get functions
GetSchemaTable method
HasRows function
overview of
testing stream for data 2nd
SqlDependency class
SqlError class
SqlErrorCollection class
SQLEv (SQL Server Everywhere Edition)
implementation
overview of
programming and maintenance
security
sharing and replication
SQL Server tools
SQLEngine
Visual Studio support
SqlException class 2nd
SqlMethodAttribute property
SqlParameter class 2nd
access rights and
best practices
default parameter values
duplicate parameter names
dynamic SQL
marking parameters
NULL values
Parameters collections
ResetDbType method
ResetSqlType method
Unicode literals
validation
overview of
sample application
Value property
what can be parameterized
SqlParameterCollection class 2nd
SqlTransaction class
SqlUserDefinedTypeAttribute (UDTs)
SSEE (SQL Server Express Edition)
administration 2nd
capabilities
common instances
configuring as server
crippled features
database deployment
default instances
deployment
eployment
independent instances
overview of 2nd
performance 2nd
replication
scalability 2nd
security 2nd
Service Packs 2nd
System Administrator's password
tools 2nd
user instances
stand-alone applications (client/server architectures)
advantages/disadvantages
building
connection management
connection pools
data binding
data entry paradigms
data retrieval
data storage limits
data validation
definition of
hybrid architectures
server-side cursors
stored procedures
temporary tables
Standard Edition (SQL Server)
Standard Edition (Visual Studio)
starting
SQL Profiler traces
SQL Server services
Visual Studio
state, managing
StateChange event (connections)
statements
ALTER PROCEDURE
batches
case-sensitive queries
CREATE PROCEDURE
CREATE RULE
DECLARE
DELETE
Imports
INSERT
generating with Query Designer
Identity values
NULL and DEFAULT values
IS NOT NULL
IS NULL
NULL values
preparing for execution
scripts
SELECT
CLR UDTs (user-defined types)
creating
creating cursors with
executing
joins
ORDER BYclause
ordering rows returned
parameters
SELECT INTO
WHERE clause
UNION
UPDATE
BLOBs
concurrency
decimal and floating point numbers
generating with Query Designer
parameter-based updates
SET clause
strings
strongly typed data columns
updating cursors
Using block
static cursors
static ports
statistics, client
stepping through code
stock-keeping units (SKUs)
stored procedures
CLR stored procedures
compared to functions
CountWords example
creating
performance
returning resultsets with SqlContext class
testing
creating
simple complex procedures
simple stored procedures
CustomerByState 2nd
CustomerUpdate
CustomerUpdate1
debugging
with breakpoints
from code
against local instances
on remote instances
ESPs (Extended Stored Procedures)
executing 2nd
managing 2nd
naming
RETURN values
sp_addextendedproperty
sp_dropextendedproperty
sp_prepexec
sp_updateextendedproperty
TableAdapters
updating tables with
streams
SqlDataReader
fetching all columns
fetching individual rows
fetching metadata
Get functions
overview of
testing for data 2nd
TDS (Tabular Data Stream)
strings
concatenating
strongly typed data structures [See also configuring, TableAdapters.]
building Web Services
consuming and testing Web Services
data columns
DataSets
exposing Web Services Data Sources in the UI
need for
SubReport item
summary reports
SUMUSD aggregate
Surface Area Configuration (SAC) tool
synchronous execution of SqlCommand objects
connection management
ExecuteNonQuery function
ExecuteReader function 2nd
CommandBehavior.CloseConnection
CommandBehavior.Default
CommandBehavior.KeyInfo
CommandBehavior.SchemaOnly
CommandBehavior.SequentialAccess
CommandBehavior.SingleResult
CommandBehavior.SingleRow
ExecuteScalar function 2nd
ExecuteXmlReader function 2nd
overview of
SqlDataReader object
best practice designs
fetching all columns
fetching individual rows
fetching metadata
Get functions
overview of
testing stream for data 2nd
Synonyms 2nd
sysname data type
System Administrator passwords
system databases
System.Data namespace
DataColumn objects
DataRow objects
DataSet objects
overview of 2nd
populating
strongly typed DataSets
DataTable objects
DataColumns
DataRows
joins
overview of 2nd
populating DataSets with
naming conventions
overview of
System.Data.SqlClient namespace
System.Data.SqlTypes namespace
SystemDataAccess setting (SqlMethodAttribute property)


T

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

T-SQL
ad hoc queries, integrating into applications 2nd
aliasing
ALTER PROCEDURE statement
BETWEEN expressions
case-sensitive queries
CLR functions, addressing
concatenated T-SQL expressions
CREATE PROCEDURE statement
CREATE RULE statement
creating queries
dates in queries
DECLARE statement
DELETE statement
executing with Run Selection feature
FETCH operator
IN expressions
INSERT statement
Identity values
NULL and DEFAULT values
IS NOT NULL statement
IS NULL statement
JOIN operator
LIKE expressions
literals in queries
NULL values
preparing statements for execution
SELECT queries
CLR UDTs (user-defined types)
creating
creating cursors with
executing
joins
limiting number of rows returned
ORDER BY clause
ordering rows returned
parameters
SELECT INTO
WHERE clause
sql_variant datatype
T-SQL debugging
from code
against local instances
on remote instances
with breakpoints
T-SQL Functions
advantages of
Inline Functions
return values
Scalar Functions
structure of
Table Functions
T-SQL Rules
binding to columns
creating
T-SQL views
UNION queries
UPDATE statement
BLOBs
concurrency
decimal and floating point numbers
generating with Query Designer
parameter-based updates
SET clause
strings
strongly typed data columns
updating cursors
table data type
Table Functions
table-valued functions (TVFs)
TableAdapter Configuration wizard (TACW)
TableAdapter objects 2nd
adding parameters to queries
binding to
with drag and drop
without drag and drop
configuring
defined
generated parameters, managing
intertable relationships, managing in Data Source designer
limitations of
overview of
parameters, configuring with Visual Studio
selecting Query methods
single-table TableAdapters
stored procedures 2nd
TACW (TableAdapter Configuration wizard)
updating tables with
TableDefinition setting (SqlMethodAttribute property)
TableMappings property (SqlDataAdapter class)
tables
aliases
check constraints
columns
binding rules to
constraints
definition of
strongly typed data columns
custom report cache tables, building
definition of
derived tables
designing
editing with Server Explorer
identifiers
indexes
INSERTED and DELETED pseudo tables
joins
Cartesian products
inner joins
outer joins
maximum size of
naming
NULL values
parent/child relationships
primary keys
changing
defining
GUIDs (globally unique identifiers)
multi-column primary keys
primary key/foreign key relationships
rows
definition of
dropping all rows
inserting
limiting number of rows returned
ordering rows returned
row-overflow storage
updating
temporary tables 2nd
unique identifiers
updating [See updates.]
Tabular Data Stream (TDS) 2nd
TACW (TableAdapter Configuration wizard)
TCP/IP (Transmission Control Protocol/Internet Protocol)
TDS (Tabular Data Stream) 2nd
team development
Team Edition (Visual Studio)
#Temp tables 2nd
tempdb database
templates
CLR UDTs (user-defined types)
Visual Studio templates
temporary tables 2nd
Terminate function
test scripts (CLR executables)
creating
executing
testing
aggregates
CLR executables
creating test scripts
executing test scripts
CLR stored procedures
for NULL values
reports 2nd
SqlDataReader streams for data 2nd
typICurrencyV2 UDT
Web Services
Text Boxes, prompting for parameter values
text data type
text files, accessing
TextBox property
threads [See also asynchronous execution of SqlCommand objects, BeginExecuteNonQuery function.]
Timeout parameter (server reports)
timeouts
Timestamp columns
timestamp data type
tinyint data type
Tool Tray
Toolbox (Visual Studio) [See also controls, BindingNavigator.]
BindingNavigator class
BindingSource class
events
methods
properties
typical data binding scenarios
DataGridView control
DataSet element
deprecated data access controls
finding controls in
ProgressBar control
Tool Tray
Tools menu commands
Import and Export Settings
Options
ToolStripPanel controls
ToString function 2nd
ToTable method
Trace Properties dialog (SQL Profiler)
traces (SQL Profiler)
evaluating
starting
transaction logs
Transaction property (SqlCommand object) 2nd
transactions
CLR executables
transaction logs
transactions-per-second limits (DBMSs)
Transmission Control Protocol/Internet Protocol (TCP/IP)
trapping
RowUpdated event
RowUpdating event
triggers 2nd
CLR triggers
creating
DDL triggers
DML triggers
INSERTED and DELETED pseudo tables
updates
TRUNCATE TABLE operator
Trusted_Connection keyword
tuning performance [See performance, CLR executable performance, evaluating.]
TVFs (table-valued functions)
type safety
typed DataSets
TypeId setting (SqlMethodAttribute property)
types [See data types, ANSI.]
typICurrencyV2 UDT
IBinarySerialize Read and Write subroutines
Parse function
private variables
testing
ToString function


U

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

UDF (User-Defined Functions)
advantages of
Inline Functions
return values
Scalar Functions
structure of
Table Functions
UDTs (user-defined types) 2nd
AddLoan
CLR UDTs
challenges of
debugging
IsNull property
methods
Null property
NULL values
Parse function
performance tuning
public properties 2nd
public structure declaration
referencing
referencing and updating values
SELECT queries
SqlUserDefinedTypeAttribute
stepping through code
templates
ToString function
typICurrencyV2 UDT
user-defined serialization
validating data
when to use
creating with Server Explorer 2nd
instantiating UDT variables
UID connection pools
underscore (_)
Unicode
compared to ANSI
data types
literals
UNION queries
UNIQUE constraint
unique identifier data type
UNSAFE setting (CLR executables)
untyped DataSets
Update Criteria property
Update method 2nd
Update Query type (Query Designer)
UPDATE statement
BLOBs
concurrency
decimal and floating point numbers
generating with Query Designer
parameter-based updates
SET clause
strings
strongly typed data columns
updating cursors
UpdateBatchSize property (SqlDataAdapter class)
UpdateCommand object
UpdateCommand property (SqlDataAdapter class)
UpdatedRowSource property (SqlCommand object) 2nd
UpdateEvent sample application
CustomerUpdate stored procedure
DataRow versions
Update application
batch mode updates
RowUpdated event, trapping
RowUpdating event, trapping
Update method
updates
batch mode updates
CLR UDT (user-defined type) values 2nd
with CommandBuilder
action commands, generating
adding to applications
capabilities of
concurrency management
wizard dependency on
complex updates with server-side logic
executing with RowUpdating event
overview of
UpdateEvent example [See application design, UpdateEvents application.]
concurrency management
Identity column values
overview of
rows
BLOBs
concurrency
decimal and floating point numbers
parameter-based updates
strings
strongly typed data columns
UPDATE SET syntax
server-side cursors
SqlDataAdapter class
with TableAdapter
example
generated parameters, managing
parameters, configuring with Visual Studio
stored procedures
triggers
UPPER function
user choices, limiting
user databases
creating
managing
User ID keyword
user instances, enabling 2nd
user settings
user-defined types [See UDTs (user-defined types).]
usernames
Using operator 2nd


V

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

validation
CLR UDTs (user-defined types)
overview of
sample application
Value property (SqlParameter class)
ValueMember property
varchar data type 2nd
variables, @@IDENTITY 2nd
VIA (Virtual Interface Adaptor)
View menu commands, Server Explorer
views
creating
Database Snapshots
indexes
managing with Query Designer
T-SQL views
Virtual Interface Adaptor (VIA)
Visual Basic
Visual Basic .NET
CLR executables
Imports statement
NULL values, testing for
Using block
Visual Studio
beta software
building ConnectionStrings
class diagrams
CLR executables
code generation
configuring parameters with
controls [See classes, BindingNavigator; controls, BindingNavigator.]
CTP (community tech preview) releases
custom configurations
creating with Options dialog
custom project templates
online or local help
saving and loading
database connections
Database Diagrams
creating
definition of
databases
creating in code
creating with Server Explorer
installing
image library
MSDN documentation
Setup program
launching
Object Browser
overview of
Query Designer
criteria pane
Delete Query type
derived tables
diagram pane
Insert Results Query type
Insert Values Query type
JOIN behavior 2nd
Make Table Query type
managing database views with
overview of
Select Query type
SQL pane
Update Query type
Report Designer
Report Items toolbox
Report Viewer
configuring
events
methods
overview of
properties
Server Explorer [See Server Explorer, assemblies, managing.]
SKUs (stock-keeping units)
SQLEv (SQL Server Everywhere Edition), support for
TableAdapter objects
tables, editing with Server Explorer
Toolbox [See also controls, BindingNavigator.]
BindingNavigator class
BindingSource class
DataGridView control
DataSet element
deprecated data access controls
finding controls in
ProgressBar control
Tool Tray
versions
Visual Studio Toolbox [See also controls, BindingNavigator.]
BindingNavigator class
BindingSource class
events
methods
properties
typical data binding scenarios
DataGridView control
DataSet element
deprecated data access controls
finding controls in
ProgressBar control
Tool Tray


W

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

WaitAll function
WaitAny function
WaitHandle objects 2nd
waiting for asynchronous operations 2nd
web application connection pools
Web Services
ASP applications versus
building
consuming and testing
defined
exposing Data Sources in the UI
.NET Framework and
XMLWeb Service architectures
WHERE clause
"white window of death,"
whiteboards
wizards
DACW (DataAdapter Configuration Wizard)
dependency on CommandBuilder
installing
dependency on CommandBuilder
Export Template Wizard
Import and Export Settings Wizard
Workgroup Edition (SQL Server) 2nd


X

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

xml data type
XML Web Service architectures
XSD files
XZoomModeproperty (Report Viewer class)


Y

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

YADAI (yet another data access interface)
Yukon [See SQL Server, action commands.]


Z

Previous Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

ZoomPercent property (Report Viewer class)