If you could give every developer a copy of the database, would you?

The simple truth is that databases can’t be given to every developer like source code.

Databases can be large, complex, and extremely difficult to copy. It is rare to see developers or even development groups that have a copy of the full production database to use for their tasks.

On the other hand, developers almost always have access to a full copy of the application code stack. Source control methods like git make it easy to clone, fork, and track code from the component level to the entire codebase.

For databases, the typical strategies are:

1. Share a full copy of the source database across a group
2. Provide small subsets of data from the source database to developers
3. Allocate schema only ‘databases’ per developer

Each of these strategies presents their own problems.

Sharing a database across a group of developers means that any change one developer makes can impact the other developers. There is no automated way to protect a group of developers from the changes of a single developer. To protect development problems that can be caused by new changes, the changes have to be reviewed. Reviews can take up to a couple of weeks, meaning the coder who wants to implement these changes has to wait. The first in a long line of delays as a result of inadequate database resources.

If developers are given subsets of the database (partial copies, only some data) or empty skeleton schemas, it will easily lead to code that has not been tested on the full range of data that is present in production. This of course opens the door to untested bugs born from outliers in the data and queries that are not optimized on the full set of data. Finding these bugs late in the development cycle is more costly by far than finding them immediately with the developer writing the code.

With Delphix you can........

Version data like code

How does it work?

According to one of our customers...

"Delphix maintains a continuous streaming backup of our production databases and allows us to create as many full copies as we want.  Copies can be masked or unmasked and are created in minutes, using minimal storage.   Delphix enables us to deliver projects faster whilst saving us time and money.”

Agile Copy Data Management

Delphix Agile Copy Data Management

Copy Data Management

Whilst there has been a gradual evolution in agile infrastructure (IaaS) and software stack provisioning architectures (PaaS), Delphix provides the missing piece in the agile architecture jigsaw by addressing Data as a Service (DaaS) through agile Copy Data Management.

Copy Data Management is generally defined as a solution which takes care of the curation, provisioning and security of all non-production database copies for Development, Analytics, Operations and Data Protection.

Networked Storage

Storage was the first component targeted by outsourcers to consolidate.  Moving from siloed, dedicated, Direct Attached Storage (DAS) to networked storage allowed overall storage consumption to be reduced by consolidating unused capacity and also allowed storage to be managed centrally.  However one of the downsides was that networked storage, typically Storage Area Networks (SANs), required a dedicated fibre channel data network to ensure performance for the high volumes of data traffic moving between database servers and the networked storage arrays.

In today’s world of agile infrastructure where resources are pooled, and allocated using virtualization technologies, having a dedicated network is an anachronism.   A dedicated resource means that unused capacity cannot be used by other workloads and the dedicated resource itself cannot expand or burst out to use capacity from elsewhere.

A move from siloed networks requires high bandwidth networks where capacity can be appropriately partitioned and allocated to accommodate multiple workloads. The throughput and capacity of Ethernet networks are evolving extremely fast allowing a single team of network administrators to manage all network traffic.  High throughput Ethernet allows the network itself to be a pooled resource with all the flexibility, agility and cost advantages that brings.

Storage Snapshots

Storage Snapshots have been around for many years and were originally designed to enable consistent backups of active database systems.  A backup takes a long time and most database systems cannot be suspended for the entire duration of a backup.  Storage snapshots allow a database to be queisced for a brief duration whilst the snapshot is taken.  The database can then be un-queisced allowing transactions to proceed and a consistent backup of the database can then be taken asynchronously via the snapshot.

Clones of snapshots can be taken to provision database copies. However supporting clones has an impact on storage performance and hence clones are usually driven off a full copy of production maintained on a separate storage server.  This mitigates the performance impact of clones on the production storage array at the cost of maintaining a replication method and additional storage.

Unfortunately the performance of cloned storage snapshots also degrades quite rapidly over time with IOPS dropping by up to 50% and latency spikes of 20ms.

Like most technologies storage snapshots are great when used for what they were originally designed for but start to disintegrate for non-primary use cases.

Virtual Data

Virtualising a shared resource is a pre-requisite for IT agility, which is why Delphix virtualizes data.  Delphix virtual data allows many users to share the same copy of data with the appearance of having their very own private, dedicated, full sized, read-write copy.  Delphix compression and de-duplication allow Delphix to store a sequential timeline of multiple database versions spanning long retention periods.  Delphix compression also increases performance by reducing physical IO requests and expands the effective size of the Delphix data cache which further reduces the number of physical IO requests.  These are two of the primary reasons why Delphix virtual data is often faster than purely physical storage based solutions.

Designed for Agility

Delphix itself is deployed as a Virtual Machine to leverage the easier deployment, management, scalability and HA of a hypervisor environment.  Almost every customer has skills in managing Virtual Machine environments.

Delphix is storage agnostic and leverages the flexibility and high performance of Ethernet to provision access to shared database versions over the network using NFS and iSCSI. 

The final key ingredient for agility is automation.  Any manual process involved in common workflows becomes a constraint and renders a solution unpredictable and non-scalable.  Delphix automates the entire data capture, curation, provisioning and versioning processes.

Access to Delphix database version control is available out-of-the box through self-service automation providing GIT like capabilities for databases of any size.

We call this Agile Copy Data Management.

Agile or Fragile

At a recent DevOps presentation by Gene Kim I was reminded of some startling facts on what differentiates high performing IT organisations from everyone else.

High performing  IT organisations do not have code freezes, in fact they make more changes during the periods that would otherwise be code freeze periods.  This seems counter intuitive, how is this even possible?   A code freeze typically occurs at peak periods of customer activity and hence no one wants to make changes which might upset the running of live applications.   Sounds sensible but what it really means is that just at the very time when changes could have the most beneficial impact you don’t make them because of a more fundamental issue, you are not able to make changes safely, in other words your code release process is fragile.

Agile development advocates that you can make changes frequently, but for that you must be able to make changes safe in the knowledge that these changes will not regress the functionality or performance of live applications.  High frequency changes also means you can experiment and prove the value of alternative solutions in production, testing not the stability of the change but the suitability of the change.

Agile development teams don’t measure releases per year but deployments per day.  A deployment is any change to any component of the application, environment or infrastructure.  Which reminds me of another startling observation, another top predictor of IT performance is “version control for all production artifacts”.  We all know that there are just as many configuration changes as there are source code changes and version control allows you to provision, compare and debug holistically.   The first question anyone asks when something breaks is “what changed?” which is relatively easy to answer for source code using source code control but is usually difficult and time consuming to answer for everything else.

So how does Delphix virtual data help with agile development?  A key to implementing agile development is making environments available early in the development cycle.  For database based applications that means provisioning a full read write copy of the database for every developer, tester and batch test activity.   A de-sensitised copy of the production database is a crucial artifact in the set of components which make up a non-production clone environment.  For most customers this is just not feasible, but with Delphix full database copies can be version controlled, can be versioned in seconds and can be provisioned or switched between versions in minutes.  Delphix allows you to manipulate and control databases versions as quickly and easily as source code files.

The example below shows how Delphix allows developers to submit regression tests on demand by automating and accelerating the data provisioning process in line with automated test suites.  Making environments available early in the development lifecycle allows a developer to get fast feedback on the success and implications of his or her proposed change.  You also get better value from your test environments as a greater ratio of time is spent running tests rather than preparing to run tests.

With more tests run more frequently and earlier in the cycle, the net result is that problems are discovered early and fixed more quickly, as illustrated below.

Virtual databases are a key enabler to Agile Development, allowing IT to produce higher quality releases, faster.  Be Agile not Fragile.

Delphix versus Storage Snapshots

Many customers ask, "What is the difference between Delphix and Storage Snapshots?" This article lists some of the key capabilities that Delphix provides over and above Storage Snapshot based cloning solutions to meet the increasing business demand for Agile Development.

First it is useful to contrast the distinct goals and implementation behind Storage Snapshots and Delphix.

Storage Snapshots

The primary use for storage snapshots is to enable backups of active database or file systems.  A backup takes a long time and most database systems cannot be suspended for the entire duration of a backup.  Storage snapshots allow a database to be queisced for a brief duration whilst the snapshot is taken.  The database can then be un-queiesced allowing transactions to proceed and a consistent backup of the database can then be taken asynchronously via the snapshot.  So a single snapshot can be used to provide a consistent view of a database.  Storage snapshot solutions also allow cloning enabling multiple snapshots.  These read write clones can be used to provision database copies quickly and with low storage overhead, consuming only changed blocks per snapshot.

Supporting clones has an impact on storage performance and hence clones are usually driven off a full copy of production maintained on a separate storage server.  This mitigates the performance impact of clones on the production storage array at the cost of maintaining a replication method and additional storage.

Delphix

Delphix is designed to support Agile Development not just backups and clones.  Agile Development requires fully automated end-to-end provisioning of database copies, ready to use running database environments and fast database manipulation operations exposed through self-service.   Like storage snapshots Delphix virtual databases can be created in minutes and each copy consumes very little incremental storage however a virtual database can be manipulated directly by the end user, typically a developer or tester, through self-service.

Aside from provisioning database copies, virtual databases support the full range of developer and tester activities such as bookmarking (saving and sharing a database state), running multiple database versions (branches), rewinding for repeated consistent tests and refreshing for operating on up-to-date data.  All of these operations are fully automated, complete in minutes and can be applied to a set of databases and optionally the application stack, and treated as a single entity.  All this is done without consuming additional database and application server resources.

Delphix synchronises with source databases with zero impact.  There is no need to replicate or copy the source databases as Delphix synchronises directly with the production database or via backups.

Accelerating Release Cycles

Delphix provides the following benefits for all Software Development Life Cycles:

1.    Manage multiple data sources as one

Many applications rely on more than one data source that must be kept synchronized. Delphix can provision and manipulate a set of databases, folders and data files as a single consistent entity.

2.    Masking and non-production data preparation

Delphix allows you to prepare a copy of production data for non-production by modifying or masking a single virtual database (VDB) once and then creating virtual copies from that single master VDB, executing the preparation phase once and enabling all downstream copies to benefit from that operation.

3.    Performance

Delphix' compression and caching reduces the workload on the underlying storage making it possible to run multiple workloads on Delphix enabled storage faster than using physical storage directly.

4.    Ease of Use

Delphix has class leading end-to-end automation covering deployment, synchronisation, administration, security, provisioning, and replication exposed through self-service user interfaces and leveraged APIs.

5.    Storage efficient

Delphix uses minimum storage when synchronising, provisioning and archiving data and is more storage efficient than storage snapshot based solutions.  All data held in Delphix is compressed and implicitly de-duplicated.

6.    Storage Agnostic

Delphix is storage agnostics allowing customers to switch storage vendors and leverage new developments in storage technologies.

7.    Provenance

Delphix has the best provenance of any Data as a Service (DaaS) solution with over 120 of the Fortune 500 companies doubling their application release frequency using Delphix.

In Summary

If you want to take backups you can use storage snapshots to give you a consistent discrete view of the data, however you could backup directly from Delphix from any point in time, avoiding impacting your production storage.

If you want to save storage you could use storage snapshots but Delphix saves even more storage and is storage independent.

If you want to support agile development through agile provisioning and agile manipulation of full sized data sets then Delphix is your only option.

Delphix is simpler, faster, more storage efficient and supports agile development more effectively than storage snapshots.

Use Delphix to virtualise Oracle database binaries for easier maintenance and improved server utilisation

Delphix is well known for its ability to virtualise and provision databases.  However Delphix has for some while also been able to virtualise folders and filesystems.

One of the most common use cases for Delphix file virtualisation (vFiles) is to virtualise application binaries.  Delphix customers have found that the Oracle database binaries stored under $ORACLE_HOME are also a great candidate for virtualising for several reasons.  We call this Virtual Oracle Home (VOH).

Maintaining and Distributing Patched Oracle Database Binaries

Typically this process involves:

1. Patch the Oracle Home with an Oracle issued PSU
2. Validate that the Patch works
3. Patch all of the affected DB Target Servers

Typical Oracle Database Patching Process

With Delphix vFiles this becomes a much more efficient two step process.

Step 1:  Virtualise and synchronise your patched Oracle Home directory.

1. Establish Patch Master and link to Delphix
2. Install patch on DB Patch Master
3. Take Delphix Snapshot to ingest patched binaries
4. Provision/Refresh Oracle Home to DEV/TEST DB server
5. Use Refresh, Rewind, Reset data control features tovtest patch, remedy errors and validate patch.

        If patch fails, you can “Rewind” the binaries to previous patch set

Step 2:  Distribute either virtual or physical copies

Now you can distribute your patched ORACLE_HOME, either as a virtual copy using NFS mounts or as a physical copy using Virtual to Physical (V2P) re-hydration of the patched ORACLE_HOME folder hierarchy on the target servers.

Other major use cases for Virtual Oracle Home (VOH)

DevOps - provision entire versioned stack

Archive - ensure exact database version is archived along with the database

Server Utilisation - make better use of your dev/test server pool by provisioning pre-requisite Oracle binaries on demand

Testing-as-a-Service - fully automated Regression Testing by provisioning all the components for your tests on any compatible server and O/S

How do I Implement Virtual Oracle Home (VOH)

Please see my upcoming blog post on implementing VOH using Delphix vFiles and hook scripts.

Learn more at Delphix Agile Data Managementwww.delphix.com