Problems with Storage Snapshot based DaaS

Storage Snapshot Cloning Architecture

Customer issues with Storage Snapshot based Cloning

Many customers have mentioned several common problems with their existing storage snapshot based cloning solutions.

To make clones from storage snapshots you must have a full copy of the source database.   It must be a copy because you don’t want to create storage clones on your production storage array and potentially compromise performance.  It must be a full copy because that’s how storage snapshots work.

The major problem here is that you need a mechanism to maintain this full master copy on a different storage array.  Using database vendor specific replication technologies will require you to establish and maintain a variety of solutions if you use more than one database vendor, so most customers use storage replication.  However storage replication does not guard against physical or logical corruption making its way to the replica copy, that might have been avoided using the database vendor replication methods.  In addition we are now locked into the storage vendor for replication, snapshots and clones.

The next issue is that the performance of storage snapshots tends to decline quite rapidly.   Some customers are forced to periodically instantiate new master full copies just to restore performance.   So now we have maintenance, availability and a storage overhead issue.

Finally most storage snapshot solutions do not have automated maintenance workflows to cover the three main requirements for DaaS which are synchronization, governance and provisioning.   For many customers this makes their storage snapshot based cloning solution non-scalable from an operational and performance point of view.

How is Delphix Different

Ø  Delphix is storage agnostic

Ø  Delphix does not require a full copy of the source database(s).

Ø  Delphix automatically synchronizes with source database and retains a compressed de-duplicated copy.

Ø  Delphix can validate the integrity of the Delphix maintained copy.

Ø  Delphix works the same for all supported databases.

Ø  Delphix reduces the workload on the underlying storage by minimizing storage IO.

Ø  Delphix automates the entire DaaS workflow end-to-end.

Ø  Delphix provides self-service interfaces for operational and end-user teams.

The Business Case for Delphix Data Masking

 

Delphix Masking

Data Masking is necessary to secure non-production copies of production data.  Since data masking is irreversible a masked database copy can be safely used by users who require access to production like database copies without compromising personal or company confidential information.

However data masking is a non-trivial task, you need to modify or generate a full sized copy of a database replacing all sensitive data with plausible replacement values.  This takes a lot of processing effort as every row in every table containing sensitive data needs to be updated or replaced.

Many customers see data masking as an expensive process which reduces agility and consequently increases project costs with little project benefit.  Of course there are an increasing number of data protection regulations due to be enforced with greater vigour.  These regulations do not change the economics but just make it more pressing to implement a data masking solution and somehow absorb the cost, they are definitely a stick and not a carrot.

Many Delphix customers are finding a way to solve the masking problem both technically and commercially.   The commercial value of Delphix data masking is to help reduce costs by enabling customers to use cloud resources for non-production activities.  Whilst it may be acceptable to run with non-masked data on premise, most of us agree that data masking is essential when operating in the cloud.

Delphix Replication makes it easy to mask data on-premise and then replicate only masked data copies to a Delphix Engine sitting, in another Data Centre, on Amazon or, in future, Azure.   Delphix Replication is surprising easy to configure and use, you basically install a second Delphix Engine to replicate to, provide the details of the replica engine to the primary engine, and then specify which masked data copies to replicate.

Of course you still have to mask the data first. Delphix customers create a virtual database (VDB) and prepare that VDB for non-production use, e.g. removing production usernames and passwords, and then obfuscate the sensitive data by running one more masking jobs against the VDB.  Running masking jobs in parallel can reduce the end-to-end masking time substantially.  Use of a VDB eliminates the need for a full database copy as the VDB will only consume storage for the data blocks modified as part of the masking process, this is typically less than 20% of the source database size.

The masked VDB can then be replicated and all non-production VDB copies use this masked master.

Delphix Masking is designed to make the process of data masking easier and faster than most masking tools. Delphix Masking helps identify potentially sensitive columns, does not require a data model to ensure referential integrity and uses the same algorithm to mask a given column, attribute or field across all data sources.

The most commonly used Delphix Masking algorithm is called Secure Lookup which is a method to replace existing values with values from a generated list of realistic but artificial values.  For example if I have a database with UK National Insurance Numbers (NINOs), I can use a script which can generate artificial NINOs according to the algorithm published at NIM39110 - National Insurance Numbers (NINOs): Format and Security.  I would then create a Secure Lookup algorithm and attribute and load this set of generated values to my Secure Lookup algorithm.  My new algorithm will replace the original values with values chosen from this list.   Since NINOs are unique I would generate a list with at least the same number of unique NINOs in the source table.   The Secure Lookup algorithm is deterministic, meaning the same input value will map to the same replacement value chosen from the generated list.  Hence if the NINO appears in more than one table but I use this algorithm against both then referential integrity is preserved.

Finally, for customers who use database encryption on their production databases, if you want to encrypt copies on non-production you would typically need to decrypt and re-encrypt using new keys to segregate from production.  Aside from this complexity, encryption licenses for non-production are expensive and unnecessary when data is masked.

Using the speed and cost savings of Delphix Provisioning, Replication, Masking and optionally cloud hosting, allows Delphix customers to solve the problem of provisioning full sized, secure copies of production efficiently and cost effectively.

Delphix versus Storage Snapshots

Many customers ask, "What is the difference between Delphix and Storage Snapshots?" This article lists some of the key capabilities that Delphix provides over and above Storage Snapshot based cloning solutions to meet the increasing business demand for Agile Development.

First it is useful to contrast the distinct goals and implementation behind Storage Snapshots and Delphix.

Storage Snapshots

The primary use for storage snapshots is to enable backups of active database or file systems.  A backup takes a long time and most database systems cannot be suspended for the entire duration of a backup.  Storage snapshots allow a database to be queisced for a brief duration whilst the snapshot is taken.  The database can then be un-queiesced allowing transactions to proceed and a consistent backup of the database can then be taken asynchronously via the snapshot.  So a single snapshot can be used to provide a consistent view of a database.  Storage snapshot solutions also allow cloning enabling multiple snapshots.  These read write clones can be used to provision database copies quickly and with low storage overhead, consuming only changed blocks per snapshot.

Supporting clones has an impact on storage performance and hence clones are usually driven off a full copy of production maintained on a separate storage server.  This mitigates the performance impact of clones on the production storage array at the cost of maintaining a replication method and additional storage.

Delphix

Delphix is designed to support Agile Development not just backups and clones.  Agile Development requires fully automated end-to-end provisioning of database copies, ready to use running database environments and fast database manipulation operations exposed through self-service.   Like storage snapshots Delphix virtual databases can be created in minutes and each copy consumes very little incremental storage however a virtual database can be manipulated directly by the end user, typically a developer or tester, through self-service.

Aside from provisioning database copies, virtual databases support the full range of developer and tester activities such as bookmarking (saving and sharing a database state), running multiple database versions (branches), rewinding for repeated consistent tests and refreshing for operating on up-to-date data.  All of these operations are fully automated, complete in minutes and can be applied to a set of databases and optionally the application stack, and treated as a single entity.  All this is done without consuming additional database and application server resources.

Delphix synchronises with source databases with zero impact.  There is no need to replicate or copy the source databases as Delphix synchronises directly with the production database or via backups.

Accelerating Release Cycles

Delphix provides the following benefits for all Software Development Life Cycles:

1.    Manage multiple data sources as one

Many applications rely on more than one data source that must be kept synchronized. Delphix can provision and manipulate a set of databases, folders and data files as a single consistent entity.

2.    Masking and non-production data preparation

Delphix allows you to prepare a copy of production data for non-production by modifying or masking a single virtual database (VDB) once and then creating virtual copies from that single master VDB, executing the preparation phase once and enabling all downstream copies to benefit from that operation.

3.    Performance

Delphix' compression and caching reduces the workload on the underlying storage making it possible to run multiple workloads on Delphix enabled storage faster than using physical storage directly.

4.    Ease of Use

Delphix has class leading end-to-end automation covering deployment, synchronisation, administration, security, provisioning, and replication exposed through self-service user interfaces and leveraged APIs.

5.    Storage efficient

Delphix uses minimum storage when synchronising, provisioning and archiving data and is more storage efficient than storage snapshot based solutions.  All data held in Delphix is compressed and implicitly de-duplicated.

6.    Storage Agnostic

Delphix is storage agnostics allowing customers to switch storage vendors and leverage new developments in storage technologies.

7.    Provenance

Delphix has the best provenance of any Data as a Service (DaaS) solution with over 120 of the Fortune 500 companies doubling their application release frequency using Delphix.

In Summary

If you want to take backups you can use storage snapshots to give you a consistent discrete view of the data, however you could backup directly from Delphix from any point in time, avoiding impacting your production storage.

If you want to save storage you could use storage snapshots but Delphix saves even more storage and is storage independent.

If you want to support agile development through agile provisioning and agile manipulation of full sized data sets then Delphix is your only option.

Delphix is simpler, faster, more storage efficient and supports agile development more effectively than storage snapshots.