If you could give every developer a copy of the database, would you?

The simple truth is that databases can’t be given to every developer like source code.

Databases can be large, complex, and extremely difficult to copy. It is rare to see developers or even development groups that have a copy of the full production database to use for their tasks.

On the other hand, developers almost always have access to a full copy of the application code stack. Source control methods like git make it easy to clone, fork, and track code from the component level to the entire codebase.

For databases, the typical strategies are:

1. Share a full copy of the source database across a group
2. Provide small subsets of data from the source database to developers
3. Allocate schema only ‘databases’ per developer

Each of these strategies presents their own problems.

Sharing a database across a group of developers means that any change one developer makes can impact the other developers. There is no automated way to protect a group of developers from the changes of a single developer. To protect development problems that can be caused by new changes, the changes have to be reviewed. Reviews can take up to a couple of weeks, meaning the coder who wants to implement these changes has to wait. The first in a long line of delays as a result of inadequate database resources.

If developers are given subsets of the database (partial copies, only some data) or empty skeleton schemas, it will easily lead to code that has not been tested on the full range of data that is present in production. This of course opens the door to untested bugs born from outliers in the data and queries that are not optimized on the full set of data. Finding these bugs late in the development cycle is more costly by far than finding them immediately with the developer writing the code.

With Delphix you can........

Version data like code

How does it work?

According to one of our customers...

"Delphix maintains a continuous streaming backup of our production databases and allows us to create as many full copies as we want.  Copies can be masked or unmasked and are created in minutes, using minimal storage.   Delphix enables us to deliver projects faster whilst saving us time and money.”

Agile Copy Data Management

Delphix Agile Copy Data Management

Copy Data Management

Whilst there has been a gradual evolution in agile infrastructure (IaaS) and software stack provisioning architectures (PaaS), Delphix provides the missing piece in the agile architecture jigsaw by addressing Data as a Service (DaaS) through agile Copy Data Management.

Copy Data Management is generally defined as a solution which takes care of the curation, provisioning and security of all non-production database copies for Development, Analytics, Operations and Data Protection.

Networked Storage

Storage was the first component targeted by outsourcers to consolidate.  Moving from siloed, dedicated, Direct Attached Storage (DAS) to networked storage allowed overall storage consumption to be reduced by consolidating unused capacity and also allowed storage to be managed centrally.  However one of the downsides was that networked storage, typically Storage Area Networks (SANs), required a dedicated fibre channel data network to ensure performance for the high volumes of data traffic moving between database servers and the networked storage arrays.

In today’s world of agile infrastructure where resources are pooled, and allocated using virtualization technologies, having a dedicated network is an anachronism.   A dedicated resource means that unused capacity cannot be used by other workloads and the dedicated resource itself cannot expand or burst out to use capacity from elsewhere.

A move from siloed networks requires high bandwidth networks where capacity can be appropriately partitioned and allocated to accommodate multiple workloads. The throughput and capacity of Ethernet networks are evolving extremely fast allowing a single team of network administrators to manage all network traffic.  High throughput Ethernet allows the network itself to be a pooled resource with all the flexibility, agility and cost advantages that brings.

Storage Snapshots

Storage Snapshots have been around for many years and were originally designed to enable consistent backups of active database systems.  A backup takes a long time and most database systems cannot be suspended for the entire duration of a backup.  Storage snapshots allow a database to be queisced for a brief duration whilst the snapshot is taken.  The database can then be un-queisced allowing transactions to proceed and a consistent backup of the database can then be taken asynchronously via the snapshot.

Clones of snapshots can be taken to provision database copies. However supporting clones has an impact on storage performance and hence clones are usually driven off a full copy of production maintained on a separate storage server.  This mitigates the performance impact of clones on the production storage array at the cost of maintaining a replication method and additional storage.

Unfortunately the performance of cloned storage snapshots also degrades quite rapidly over time with IOPS dropping by up to 50% and latency spikes of 20ms.

Like most technologies storage snapshots are great when used for what they were originally designed for but start to disintegrate for non-primary use cases.

Virtual Data

Virtualising a shared resource is a pre-requisite for IT agility, which is why Delphix virtualizes data.  Delphix virtual data allows many users to share the same copy of data with the appearance of having their very own private, dedicated, full sized, read-write copy.  Delphix compression and de-duplication allow Delphix to store a sequential timeline of multiple database versions spanning long retention periods.  Delphix compression also increases performance by reducing physical IO requests and expands the effective size of the Delphix data cache which further reduces the number of physical IO requests.  These are two of the primary reasons why Delphix virtual data is often faster than purely physical storage based solutions.

Designed for Agility

Delphix itself is deployed as a Virtual Machine to leverage the easier deployment, management, scalability and HA of a hypervisor environment.  Almost every customer has skills in managing Virtual Machine environments.

Delphix is storage agnostic and leverages the flexibility and high performance of Ethernet to provision access to shared database versions over the network using NFS and iSCSI. 

The final key ingredient for agility is automation.  Any manual process involved in common workflows becomes a constraint and renders a solution unpredictable and non-scalable.  Delphix automates the entire data capture, curation, provisioning and versioning processes.

Access to Delphix database version control is available out-of-the box through self-service automation providing GIT like capabilities for databases of any size.

We call this Agile Copy Data Management.

Agile Archive

When most customers install Delphix they typically add source databases by ingesting an up-to-date backup and then establishing a timeflow of continuous changes from there, however it is possible to ingest existing backups stretching back for as long as you require.

Why would you want to ingest older backups you have already into Delphix?

There are several challenges managing backup archives, one of the biggest being trying to use them.

1.     Unless you know the required date how do you know which backup to restore, assuming you are not simply restoring production from the latest backup?

2.     The time taken to restore involves copying the files and applying recovery, usually this takes a while and requires time from Database and maybe Storage Administrators.

3.     How can you be sure you can restore from a given backup unless the restore process has been tested.

4.     In many cases access to older backups is only required for reporting and compliance reasons.  These are generally not heavy duty tasks so why restore an entire physical copy for a relatively lightweight use case?

Using Delphix as an Agile Archive

1.     The Delphix ingest process will implicitly test that your backups are valid.

2.     You can provision a Virtual Database (VDB) in minutes from any of your ingested backups.

3.     You could build a catalogue of contents of each backup by starting a VDB against the oldest version, querying the content and refreshing the VDB to the next version for each backup in turn.  You could add this metadata to your version control system.

4.     You can publish those backups via self-service to anyone who may need to access, query or run a report against an archived backup.

5.     You could optionally delete your backups once copied into Delphix as Delphix will retain them for as long as required, store them much more economically, using block de-duplication and compression, and can protect them via replication, or backup.

The process to use Delphix for Agile Archiving is very straightforward.

Using SQL Server as an example you would create a Delphix dSource specifying the initial load as a specific full backup, you would then enter the backup UUID of the oldest backup you want to ingest.

For example:

  

Then you can use the Delphix Command Line Interface (CLI) to ingest a set of backups in chronological sequence, starting from the oldest.  An example command would be:

database "AdventureWorksLT2008R2" sync; set type=MSSqlSyncParameters; set loadFromBackup=true; set backupUUID="4688F509-33E1-404D-A692-F9A0373CEF75"; commit

The UUIDs can refer to full or incremental backups which you query from SQL Server.  You can run several of these CLI commands from a single script and you will see in the Delphix UI a set of discrete timecards corresponding to each ingest.

Once your ingest sequence has completed Delphix will automatically stay in synchronization with the source database when new backups, optionally including transaction log backups, are taken.

A similar approach can be used with other database types.  For example for SAP ASE (Sybase) the CLI script to ingest a sequence of archived backups might look like:

database "pubs3" update sourcingPolicy; edit sourcingPolicy; set logsyncEnabled=false; commit

database "pubs3" sync; set type=ASESpecificBackupSyncParameters; set backupFiles=pubs3.full.10293; commit

database "pubs3" sync; set type=ASESpecificBackupSyncParameters; set backupFiles=pubs3.full.31019; commit

database "pubs3" sync; set type=ASESpecificBackupSyncParameters; set backupFiles=pubs3.full.31499; commit

database "pubs3" sync; set type=ASESpecificBackupSyncParameters; set backupFiles=pubs3.full.32083; commit

database "pubs3" sync; set type=ASESpecificBackupSyncParameters; set backupFiles=pubs3.full.34057; commit

database "pubs3" sync; set type=ASESpecificBackupSyncParameters; set backupFiles=pubs3.full.34071; commit

database "pubs3" update sourcingPolicy; edit sourcingPolicy; set logsyncEnabled=true; commit

In this example we disabled and enabled LogSync before and after the ingest sequence.

Delphix Agile Archiving provides faster and more economical access to validated, archived backups and guarantees these can be accessed near instantly and on demand.