Monday, 19 January 2015

What are naming standards/conventions for SQL Server? (1/5)

I think it will be good idea to setup some naming standards throughout your database servers’ farm. Most of us already have these standards in place. And if there is nothing, I understand that it’s not going to be an easy task to implement standards, especially within an existing environment. BUT, it’s never too late. Let’s have something now.


 


Server name – <Three or Four Characters for Datacenter location> + <Two, Three or may be Four characters for Server type> + <three or four digit number>


Location example: USA, IND, DEL, LON, ZUR, NYK etc


Server Type example: SQLS, SSIS, SSRS, SSAS, ETL


So, server name should look something like – DELSSRS001 (or DEL-SSRS-001), by looking at the name only, anyone can now understand that this server is for SSRS & located at Delhi (DEL) datacenter. Additionally, you can define standards for named SQL instances.


Database Name – Naming database is a crucial task, it’s like finding a good name for a new born baby. This can be good idea to associate database name with the applications as family name. Like, <AppName> + <Underscore> + <Three Characters for Environment> + <version number>


Example: MyApp_Prd1, MyApp_Dev1, MyApp_UAT2


Once database name is set, it comes to Database Files & File groups. Database files should follow the same logic as database name. So, it will become something like


<databasename> + <Underscore> + <dat> + <count number>.mdf


<databasename> + <Underscore> + <log> + <count number>.ldf


<databasename> + <Underscore> + <dat> + <count number>.ndf


Example: MyApp_Prd1_dat1.mdf, MyApp_Prd1_dat2.ndf, MyApp_Prd1_log1.ldf


File Groups : <fg> + <Underscore> + <purpose1> + <purpose2> + <count number>


Example: fg_data1, fg_myindex_1


Stay tuned for upcoming posts on this series.




What are naming standards/conventions for SQL Server? (1/5)

Wednesday, 14 January 2015

Common Human Errors as DBA (Part 3)

 


In continuation of the series of “Common Human Errors as DBA” – here I am posting Third Part of it. You may check others here. Part 1 & Part 2


Poor naming standards & Lack of documentation: This is a common mistake (not human error) that frequently happens. From the very beginning, a DBA must setup naming standards and must communicate ‘standards’ to all stakeholders. Setting up an environment which has all naming standards in place is very easy to manage and helps others to understand quickly. Missing control over naming standard can be considered as Human Error though. So, to prevent,


  • You may create a scheduled report to show the list of objects which are not meeting the standards.

  • You must communicate your standards; especially to Developers

  • You must not allow exceptions when it comes to standards

Added indexes to every column in a table for best performance: This can be consider as human error that without evaluating existing indexes on other columns, during a course of time, a DBA created indexes on all (or most of the) columns. And top of it, DBA forgot that indexes need maintenance too. DBA needs to be careful about index selection and their maintenance. I could not figure out any prevention method for this. I can say, “Be careful as always”. However, if you have some suggestions, please feel free to add comments below in the post.


Gave database user DBO or SA rights: Sometimes, in non-production environment, while troubleshooting issues, DBA may need to elevate access for a database user. Once DBA finds the root cause, DBA must revoke elevated access as quickly as possible. I have seen some cases where DBA felt excited after fixing those issues and forgot to revoke access. This eventually caused issues at the end of the day. To prevent this,


  • A weekly Audit report can be created and delivered to DBAs for review.

  • An alerting system can be created if there is any change with permissions.

  • Changes should be tracked in Incident or Change documentation.

 


Hopefully this series helped you. Stay tuned.



Common Human Errors as DBA (Part 3)

Tuesday, 13 January 2015

SQL DBA Cheatcode Part 1 : Few Important Queries for any DBA

Hi Friends,


From last few days, I was thinking about writing a series on Queries\T-SQL commands that we use in our day to day operations or those queries which should be handy for any SQL DBA(Junior or senior) out there. In this series, I’ll try to share as many queries as I have or I use. There are 1000s features in SQL Server and each feature has its own set of infinite queries so any blog cannot cover all queries and all features but I will try to cover the queries that is needed in our day to day activities as a DBA.


Comments are most welcome by readers which can also enhance my database of queries. So here goes first part of this series.

**DISCLAIMER : I have personally used all these queries but these may or may not work in your environment depending on your server setting, server edition\version etc. Please test these queries in lower environment first before executing these in production directly**


Note :- I would like to thank countless sites and articles present there on google which helped me save all these queries to my personal QUERY DATABASE. This series is for knowledge sharing pupose only and you can share these queries again to your colleagues and friends.


QUERY 1 :-
Free & Used space of all files in a Database : Many times we need to check which all datafiles are using how much space. Instead of using GUI, you can use below query directly to get all the details :-


select

name

, filename

, convert(decimal(12,2),round(a.size/128.000,2)) as FileSizeMB

, convert(decimal(12,2),round(fileproperty(a.name,’SpaceUsed’)/128.000,2)) as SpaceUsedMB

, convert(decimal(12,2),round((a.size-fileproperty(a.name,’SpaceUsed’))/128.000,2)) as FreeSpaceMB

from dbo.sysfiles a


QUERY 2 :-
All Database files in a Instance with location : This query lists all the database files present in a Instance. Output of this file contains Logical filename and its physical path in the server.


SELECT name, physical_name AS current_file_location

FROM sys.master_files


QUERY 3 :-


Query to find all the processes running on a particular DB : In many cases, you may not want to see for running processes on all the databases from sp_who2 and want to look directly for a particular database :-


SELECT [Database]=DB_NAME(dbid), spid, last_batch, status, hostname, loginame

FROM sys.sysprocesses

WHERE dbid = DB_ID(‘TESTDB’); —-change the databse name here


QUERY 4 :-


All Databases last access time from last reboot :- You may need to find which database is not being used by users in long time. This query may come handy but it has limitations that it contains data from last service restart only as data from DMV dm_db_index_usage_stats gets flushed after every service recycle :-


SELECT DatabaseName, MAX(LastAccessDate) LastAccessDate

FROM

(SELECT DB_NAME(database_id) DatabaseName ,

last_user_seek ,

last_user_scan ,

last_user_lookup ,

last_user_update FROM sys.dm_db_index_usage_stats) AS PivotTable UNPIVOT

(LastAccessDate FOR last_user_access IN

(last_user_seek ,

last_user_scan ,

last_user_lookup ,

last_user_update) )

AS UnpivotTable GROUP BY DatabaseName

HAVING DatabaseName NOT IN (‘master’, ‘tempdb’, ‘model’, ‘msdb’)

ORDER BY 2


QUERY 5 :-


Script to find fragmentation level for every table of a database :- The below script provides fragmentation percentage for every index of every table in a database :-


select TableName=object_name(dm.object_id)

,IndexName=i.name

,IndexType=dm.index_type_desc

,[%Fragmented]=avg_fragmentation_in_percent

from sys.dm_db_index_physical_stats(db_id(),null,null,null,’sampled’) dm

join sys.indexes i on dm.object_id=i.object_id and dm.index_id=i.index_id

order by avg_fragmentation_in_percent desc


I shall be back with more queries soon. Please comment your views. Thanks for reading!!



SQL DBA Cheatcode Part 1 : Few Important Queries for any DBA

Saturday, 10 January 2015

How to get access on SQL Server if you don"t have "sa" password or sysadmin access?

PROBLEM


DBAs need to have sysadmin access on SQL Server to support Databases effectively. There may come situations where DBAs neither have sysadmin access nor [sa] password. In that case, DBA needs to perform below activity to get the access.


RESOLUTION


PREREQUISITES:


1) Administrative privileges needed on Host Server.

2) Downtime of approximately 10 minutes needed because SQL Server would be started in ‘Single User’ mode.


PROCEDURE:


1. RDP to the server with the account which has ‘Admin’ rights on the Hosting Windows box. Using a service account is good idea.


2. Open ‘SQL Server Configuration Manager’ :


Start>All Programs>Microsoft SQL Server 2008 (or 2005)>Configuration Tools>SQL Server Configuration

Manager


3. Start SQL Services in ‘Single User Mode:


  • Right Click on SQL Services and click ‘properties’

  • Go ‘Advanced’ tab then open ‘Start up Parameters’

  • Write ‘-m’ at the end, then click ‘OK’.

This would open a dialog box informing that SQL Services need to be restarted for this. Click ‘OK’.


  • Stop dependent services like SQL Agent. Restart SQL Services.

4. Connect to SQL instance using SQLCMD


  • Right Click on Command prompt, select ‘Run as Administrator’ (in Windows 2003 and lower, run as administrator is not available and not required)

  • Now connect to SQL instance using SQLCMD, write below command:

  • SQLCMD –E –S <instance name>

5. Now create a windows login (i.e. group account used by DB team), use below command:


CREATE LOGIN [Login_name] FROM WINDOWS WITH DEFAULT_DATABASE=[master]

GO


6. Now grant ‘sysadmin’ access to this newly created login, use below command:


EXEC master..sp_addsrvrolemember @loginame = "login_name", @rolename = "sysadmin"

GO


7. Restart SQL Services in multi user mode. For doing that, remove ‘-m’ from Startup parameters and restart SQL

Services.


Now DBA gets sysadmin access at SQL Server with the above created Login Name.




How to get access on SQL Server if you don"t have "sa" password or sysadmin access?

Wednesday, 7 January 2015

Common Human Errors as DBA (part 2)

Eliminating Human Error


You must have read my previous post (Part-1) about Common Human Errors as DBA and how to avoid them. Here is another post of the same series. Third error –
Data or Log file Shrinking – sometimes, one may not be checking how much free space is available in data or log files and shrinking databases again and again. Or setting ‘Auto Shrink’ database option to true is equally harmful; this is harmful especially in high transactions environments or with large databases. Remember, if you grow your database by small increments or if you grow it and then shrink it, you can end up with disk fragmentation. Disk fragmentation can cause performance issues in many circumstances. So, to avoid this, you must always have monitoring in place. You can use custom scripts or any other monitoring tool to achieve this task.


Setting database recovery option to ‘Full’ – A common mistake is when recovery option of a database (or Model database) is set to ‘Full’. And a DBA is never taking transaction log backups. In such cases the transaction log file size grows 100 times to actual data due to backup negligence. As a DBA, you must ensure that database recovery model is selected as per or according to requirements. And a backup policy must be planned accordingly. So, to prevent this you can create some weekly or daily checks, some reports, and some job alert.


I will try to post some more Human Errors in last and part 3. Stay connected!


Now, you can follow DBAThings.com on Tweeter at @DBATHINGS



Common Human Errors as DBA (part 2)

Tuesday, 6 January 2015

MAXDOP - What is best approach to use it!!!

The Maximum Degree of Parallelism (MAXDOP) setting will determine the degree to which SQL server will attempt to parallel process transactions which are received from the scheduler. As a general rule of thumb, OLTP environments with high transaction volume and repetitive queries should normally have a MAXDOP value of 1, as any benefit gained from parallel process would quickly be lost in overheads introduced in managing this. OLAP environments however (such as analysis services / data warehousing etc) where transactions are generally much fewer, however considerably more complex – these environments would benefit from a MAXDOP setting of 0 (unlimited) or near the number of CPUs in the system.


The SQL Server query optimizer does not use a parallel execution plan for a query if any one of the following conditions is true:


1 The serial execution cost of the query is not high enough to consider an alternative, parallel execution plan.

2 A serial execution plan is considered faster than any possible parallel execution plan for the particular query.

3 The query contains scalar or relational operators that cannot be run in parallel. Certain operators can cause a section of the query plan to run in serial mode, or the whole plan to run in serial mode.


The MAXDOP value can be updated during run time by the following command, noting that the value below should be adjusted to suit your requirements;


sp_configure ‘show advanced options’, 1;

GO

RECONFIGURE WITH OVERRIDE;

GO

sp_configure ‘max degree of parallelism’, 1;

GO

RECONFIGURE WITH OVERRIDE;

GO


You may also want to clear the wait statistics after this to ensure your next checks on the system are clear / fresh statistics since this change has come into effect;


DBCC SQLPERF(“sys.dm_os_wait_stats”,CLEAR);


A good indicator on whether this value has been correctly configured is the CXPACKET performance counter. Where more than 5% of the total wait conditions are due to this counter, it’s most likely that the MAXDOP value will need to be adjusted.


When all is said and done, managing your MAXDOP and cost of parallelism settings are very much part of the DBA art form, where you will need to test and evaluate your workload to find the settings that are appropriate for your server. Here’s a couple steps to get started with on your server:


1 What kind of queries and how many transactions does my server see? Talk to your developers about the processes and take a look at your Batch Requests/second perfmon counter to help you evaluate this.

2 Check your current MAXDOP and cost of parallelism settings on your server. Are they set to the defaults?

3 What do your CXPACKET waits look like relative to other waits on your server? High CXPACKET waits might be an indication that the MAXDOP on your server is set to high. To find out your waits, Glenn Berry has an excellent DMV query within his SQL Server 2008 Diagnostic Information Queries that will provide this information for you.


Once you have this information, you can make a plan to adjust your parallelism settings, but you’ll need to test any changes to your settings before implementing them.



MAXDOP - What is best approach to use it!!!

Database Engine Instances (SQL Server): A Quick Refresher

SQL Server instance of the Database Engine is nothing but a copy of the sqlservr.exe executable that runs as an operating system service. This means for each running SQL Server instance you must see one sqlservr.exe in task manager. Like in screenshot.


SQL-Instance


  • An instance manages few system databases and one or more user databases.

  • A Server/Computer can run multiple instances

  • There can be only one instance as default instance. The default instance has no name. Other Instances are called Named Instances

  • A connection request must specify both the computer name and instance name in order to connect to the instance. Default instance works with (.) as well.

Instance ID – By default, instance name is used as the Instance ID. Installation directories and registry keys are identified by Instance ID. For a default instance, the instance name and instance ID will always be MSSQLSERVER.


Starting SQL Server instance using command line


You must locate sqlservr.exe on your server. By default, sqlservr.exe is located at C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\Binn. If there are more than one instance running, another copy of sqlservr.exe is located in a directory such as C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\Binn. You may start one instance of SQL Server by using sqlservr.exe from another instance, SQL Server will start the version of the incorrect instance as well, including service packs, and this may lead to some unexpected results. To avoid, always use the correct & similar version sqlservr.exe to bring Sql server up.


You may use some commands like below, from a command prompt:


  • Run sqlservr.exe to start default instance

  • Run sqlservr.exe -s <instancename> to start named instance.

Renaming SQL Server Instance


The general recommendation is that you can change an instance name if it is the default instance, but modifying an existing named instance is not recommended. One should always uninstall and install. However in case if you need to rename – you can use below procedures:


Open Management Studio, connect to SQL server and then run


SELECT @@servername

EXEC master.dbo.sp_dropserver "MACHINENAME\OLDINSTANCENAME"

go

EXEC master.dbo.sp_addserver "MACHINENAME\newINSTANCENAME", "local"

Go

Restart SQ server services and you should see the new name. I hope this refreshes your concepts about SQL Server instance.



Database Engine Instances (SQL Server): A Quick Refresher

Monday, 5 January 2015

Fragmentation of Log files and VLFs

In a SQL server, having multiple log files does not have any performance benefits as the log files are written sequentially. The log files can have external (disk level) fragmentation or internal (VLF, Virtual Log File) fragmentation. The external fragmentation can be solved by de-fragmenting the disk on which the log file resides on, followed by properly sizing the log files, to prevent further fragmentation.


Internal fragmentation is caused by the growing and shrinking of the log files. There is a nice article by Thomas LaRock on how to use powershell to get an output of the number of VLFs for your DBs. A basic rule of thumb that I use is, if there are more than 50 VLFs for a DB, it is time to truncate it. Using dbcc loginfo(‘db_name‘) with no_infomsgs will return one row for each VLF for db_name. To ‘de-fragment’ the VLFs, do the following:


  1.  Backup the log file (when in full mode), or issue a checkpoint (when in simple mode).

  2. Truncate the log file: DBCC SHRINKFILE (‘db_logfile_logical_name‘). Should be run in the context of the DB concerned.

  3. Size the log file appropriately. ALTER DATABASE ‘db_name‘ MODIFY FILE (‘db_log_file_logical_name‘,new_size_as_int)

Do a DBCC loginfo again to see the reduced number of VLFs you now have.



Fragmentation of Log files and VLFs

Common Human Errors as DBA (part 1)

Human Error“Human Error” is not a new word. Most of the time if there is screwed up done by any one – these golden words are used. For sure, these are the Human Errors and its hard to control. Here I am trying to consolidate some common human errors that are done by DBAs at different level. As I said, its hard to control human errors but it’s always good to have a plan to avoid or recover from that situation. Simplest plan to avoid Human Error is to turn Human into Robots. Looks funny, I mean it. Guide and force DBAs to use checklists for each and every activity. Automate most of the processes which require human interventions.


Here are two good examples of Human Errors as a DBA. I will add more in Part-2


 


 


  1. DBA started Restore without taking backup of existing users: I will keep this mistake on top of chart. This is very common for any DBA, especially if someone has joined new in the team. To prevent:
    • You must focus on Automation of restore tasks. Let’s say, try building the system which does the restore for you. That’s how, you will ensure that even by mistake, step is not skipped.

    • You can try writing a check list of steps and you must follow them.


  2. Ran a long running script from local or VPN and lost connection – Another common mistake that is done, a long script (DML or DDL) started from the local machine or VPN connection and then LAN connection is lost. Script had been executed partially and had made the changes (to structure or data). Now, to prevent:
    • You must try to run the script from a server itself.

    • You must have transactions in place. If you don’t see transaction in your script, add it.


I will add few more Human Errors in the list in part 2. Stay Tuned.



Common Human Errors as DBA (part 1)

Hide Your Real Data - Data masking/ Data Scrambling/ Data Obfuscation

ch008-uf001Data masking/ Data Scrambling/ Data Obfuscation – is the task for hiding actual data before sharing it with Users, UAT Testers, Vendors, Developers or anyone else who MUST NOT have access to production data or real time data. A DBA must proactively take care of this task if no one informed about the requirements. Sensitive personal or financial information must be masked. Talking to stake holders proactively is always good before you start data masking.


How to achieve this?


There are many ways of achieving. However, this is the simplest approach of masking data. Masking data requirements differ all the time. You need to be careful about your approach.


With this approach, in nutshell, you must create a function which replaces the actual string with same size strings or replaces some characters with others. You must create a procedure which accepts Table Name and Column Name with in arguments. Then it’s purely your choice how would you like to proceed. You may create Insert trigger, Update Trigger or SQL job or OS Job to call SP for any Event etc.


Steps to Mask:


Step 1 : Create the Function  [dbo].[ ufnMaskRealData]  on the targeted Database


———–Copy from here —————————–


/****** Object:  UserDefinedFunction [dbo].[ufnMaskRealData]    Script Date: 01/05/2015 12:10:49 ******/


SET ANSI_NULLS ON


GO


SET QUOTED_IDENTIFIER ON


GO


— =============================================


— Author:        Sachin Diwakar


— Create date: 24 Dec 204


— Description:   Once called within select statement or SP – replaces the characters with the one listed in replace statment.


— =============================================


create function [dbo].[ufnMaskRealData]


(


@String varchar(255)


)


returns varchar(255)


as


BEGIN


select @String = replace(replace (replace (replace(replace( replace(replace(replace(replace(replace(replace(replace(replace(replace(replace(replace(replace(replace(@String,’o’,’e’),’a’,’o’),’i’,’a’),’u’,’i’),’t’,’p’),’c’,’k’),’d’,’t’),’e’,’x’),’o’,’r’),’l’,’s’), ‘1’,’3′),’2′,’4′),’3′,’5′),’4′,’6′), ‘0’,’9′), ‘9’,’1′),’h’,’p’), ‘k’,’n’)


return @String


END


—-go


———————-Copy end —————————————-


Step 2: Create the SP [dbo].[ uspDataMask] on the targeted Database


———–Copy from here —————————–


/****** Object:  StoredProcedure [dbo].[uspDataMask]    Script Date: 01/05/2015 12:13:39 ******/


SET ANSI_NULLS ON


GO


SET QUOTED_IDENTIFIER ON


GO


— =============================================


— Author:        Sachin Diwakar


— Create date: 23 Dec 204


— Description:   This SP masks the data in column which is passed as an argument


— Example – Exec [dbo].[uspDataMask] TableName, ColumnToBeMasked


— =============================================


CREATE PROCEDURE [dbo].[uspDataMask]


@objectName as sysname, @colname as sysname


AS


BEGIN


Declare @sql as varchar(max)


set @sql = ‘update ‘ + @objectName + ‘ set ‘ + @colname + ‘= dbo.[ufnMaskRealData] (‘+ @colname +’)’


exec (@sql)


–PRINT @sql


END


GO


————–Copy ends here ————–


Step 3: Based on requirements, you can create below trigger on the table(s) and columns(s) that you would like to be masked.


———–Copy from here —————————–


/****** Object:  Trigger [dbo].[trgDataMasktblAddress]    Script Date: 01/05/2015 12:15:17 ******/


SET ANSI_NULLS ON


GO


SET QUOTED_IDENTIFIER ON


GO


— =============================================


— Author:        Sachin Diwakar


— Create date: 24 Dec 2014


— =============================================


CREATE TRIGGER [dbo].[trgDataMaskOnYourTable1]


ON  [dbo].[ YourTable1]


AFTER INSERT


AS


BEGIN


— SET NOCOUNT ON added to prevent extra result sets from


— interfering with SELECT statements.


SET NOCOUNT ON;


begin tran t1


exec [uspDataMask] ‘dbo.YourTable’, ‘YourCol8′


exec [uspDataMask] ‘dbo.YourTable1′, ‘YourCol7′


exec [uspDataMask] ‘dbo.YourTable2′, ‘YourCol6′


exec [uspDataMask] ‘dbo.YourTable3′, ‘YourCol5′


exec [uspDataMask] ‘dbo.YourTable4′, ‘YourCol4′


exec [uspDataMask] ‘dbo.YourTable5′, ‘YourCol3′


exec [uspDataMask] ‘dbo.YourCol16′, ‘YourCol2′


commit tran  t1


END


————–Copy ends here ————–


Additionally you can create a job to mask the data as and when required. To do so, you just need to create a SQL server job and call the SP USPDataMask with the table and column name that you would like to mask.


exec [uspDataMask] ‘dbo.YourTable1, ‘YourCol1′



Hide Your Real Data - Data masking/ Data Scrambling/ Data Obfuscation

Sunday, 4 January 2015

Database Accidents : Unwanted Restore

Number of times, while working as a DBA many accidents happen. Most common of them is when a database is restored without checking if a backup is available or not. As a thumb rule – a DBA must take backups of everything before performing any action. At any point of time, I must be in position of taking things back to the previous state. I know its difficult to achieve this all times.


Lets talk about the scenario when accidentally you start restoring a database and later you realize that it wasn’t required. Now, you start to look for the backup of victim database so that you can at least bring the database to the state where it was at the time of taking last backup. And UNFORTUNATELY – the backup isn’t available. Now this is a road block and panic situation. Isnt it? I am trying to list some steps that can be taken. Remember, all situations are different and its hard to generalize any solution. However, try these things


Keep communicating to your stakeholders about the situation. Remember, this is a critical situation and communication is the key. Get consent from you stakeholders that there will be data loss for sure.


  1. Look for any other available backup of victim database.

  2. You may check if you have received any request in past which asked you to keep the backup at safe location

  3. You may check in your DR Server/Network if the backup is available.

  4. You may check in your other environment if you are copying or restoring or shipping this database.

If nothing works, there is no harm trying this solution. I have not done this personally as of now but this works in theory. However, I am going to try this for sure soon and will keep you all posted.


  • Detach the database from SQL server instance

  • Copy the corrupt data and transaction log files to a different location

  • Make sure to delete the original corrupt files

  • Create a new database with the exact same file layout, names and file sizes as the detached corrupt database

  • Stop the SOL Server instance

  • Overwrite the data and transaction log files of the newly created database with the original corrupt files

  • Start the SOL Server instance

Now that the corrupt databases is back online, it’s time to recover what data you can. To do so:


  • Execute ALTER DATABASE your_database_name SET EMERGENCY; to put the corrupt SQL database into EMERGENCY mode

  • In order to enable recovery you need to put the database into SINGLE_USER mode as well. To do so, execute ALTER DATABASE your_database_name SET SINGLE_USER;

  • Execute DBCC CHECKDB (your_database_name, REPAIR_ALLOW_DATA_LOSS) WITH NO_INFOMSGS

This allows that you have database online now. Depending on the situation – you should be able to see some of the data. Play around and see if this makes your little easy in this tough situation.



Database Accidents : Unwanted Restore