Agile Archive

When most customers install Delphix they typically add source databases by ingesting an up-to-date backup and then establishing a timeflow of continuous changes from there, however it is possible to ingest existing backups stretching back for as long as you require.

Why would you want to ingest older backups you have already into Delphix?

There are several challenges managing backup archives, one of the biggest being trying to use them.

1.     Unless you know the required date how do you know which backup to restore, assuming you are not simply restoring production from the latest backup?

2.     The time taken to restore involves copying the files and applying recovery, usually this takes a while and requires time from Database and maybe Storage Administrators.

3.     How can you be sure you can restore from a given backup unless the restore process has been tested.

4.     In many cases access to older backups is only required for reporting and compliance reasons.  These are generally not heavy duty tasks so why restore an entire physical copy for a relatively lightweight use case?

Using Delphix as an Agile Archive

1.     The Delphix ingest process will implicitly test that your backups are valid.

2.     You can provision a Virtual Database (VDB) in minutes from any of your ingested backups.

3.     You could build a catalogue of contents of each backup by starting a VDB against the oldest version, querying the content and refreshing the VDB to the next version for each backup in turn.  You could add this metadata to your version control system.

4.     You can publish those backups via self-service to anyone who may need to access, query or run a report against an archived backup.

5.     You could optionally delete your backups once copied into Delphix as Delphix will retain them for as long as required, store them much more economically, using block de-duplication and compression, and can protect them via replication, or backup.

The process to use Delphix for Agile Archiving is very straightforward.

Using SQL Server as an example you would create a Delphix dSource specifying the initial load as a specific full backup, you would then enter the backup UUID of the oldest backup you want to ingest.

For example:

  

Then you can use the Delphix Command Line Interface (CLI) to ingest a set of backups in chronological sequence, starting from the oldest.  An example command would be:

database "AdventureWorksLT2008R2" sync; set type=MSSqlSyncParameters; set loadFromBackup=true; set backupUUID="4688F509-33E1-404D-A692-F9A0373CEF75"; commit

The UUIDs can refer to full or incremental backups which you query from SQL Server.  You can run several of these CLI commands from a single script and you will see in the Delphix UI a set of discrete timecards corresponding to each ingest.

Once your ingest sequence has completed Delphix will automatically stay in synchronization with the source database when new backups, optionally including transaction log backups, are taken.

A similar approach can be used with other database types.  For example for SAP ASE (Sybase) the CLI script to ingest a sequence of archived backups might look like:

database "pubs3" update sourcingPolicy; edit sourcingPolicy; set logsyncEnabled=false; commit

database "pubs3" sync; set type=ASESpecificBackupSyncParameters; set backupFiles=pubs3.full.10293; commit

database "pubs3" sync; set type=ASESpecificBackupSyncParameters; set backupFiles=pubs3.full.31019; commit

database "pubs3" sync; set type=ASESpecificBackupSyncParameters; set backupFiles=pubs3.full.31499; commit

database "pubs3" sync; set type=ASESpecificBackupSyncParameters; set backupFiles=pubs3.full.32083; commit

database "pubs3" sync; set type=ASESpecificBackupSyncParameters; set backupFiles=pubs3.full.34057; commit

database "pubs3" sync; set type=ASESpecificBackupSyncParameters; set backupFiles=pubs3.full.34071; commit

database "pubs3" update sourcingPolicy; edit sourcingPolicy; set logsyncEnabled=true; commit

In this example we disabled and enabled LogSync before and after the ingest sequence.

Delphix Agile Archiving provides faster and more economical access to validated, archived backups and guarantees these can be accessed near instantly and on demand. 

Agile or Fragile

At a recent DevOps presentation by Gene Kim I was reminded of some startling facts on what differentiates high performing IT organisations from everyone else.

High performing  IT organisations do not have code freezes, in fact they make more changes during the periods that would otherwise be code freeze periods.  This seems counter intuitive, how is this even possible?   A code freeze typically occurs at peak periods of customer activity and hence no one wants to make changes which might upset the running of live applications.   Sounds sensible but what it really means is that just at the very time when changes could have the most beneficial impact you don’t make them because of a more fundamental issue, you are not able to make changes safely, in other words your code release process is fragile.

Agile development advocates that you can make changes frequently, but for that you must be able to make changes safe in the knowledge that these changes will not regress the functionality or performance of live applications.  High frequency changes also means you can experiment and prove the value of alternative solutions in production, testing not the stability of the change but the suitability of the change.

Agile development teams don’t measure releases per year but deployments per day.  A deployment is any change to any component of the application, environment or infrastructure.  Which reminds me of another startling observation, another top predictor of IT performance is “version control for all production artifacts”.  We all know that there are just as many configuration changes as there are source code changes and version control allows you to provision, compare and debug holistically.   The first question anyone asks when something breaks is “what changed?” which is relatively easy to answer for source code using source code control but is usually difficult and time consuming to answer for everything else.

So how does Delphix virtual data help with agile development?  A key to implementing agile development is making environments available early in the development cycle.  For database based applications that means provisioning a full read write copy of the database for every developer, tester and batch test activity.   A de-sensitised copy of the production database is a crucial artifact in the set of components which make up a non-production clone environment.  For most customers this is just not feasible, but with Delphix full database copies can be version controlled, can be versioned in seconds and can be provisioned or switched between versions in minutes.  Delphix allows you to manipulate and control databases versions as quickly and easily as source code files.

The example below shows how Delphix allows developers to submit regression tests on demand by automating and accelerating the data provisioning process in line with automated test suites.  Making environments available early in the development lifecycle allows a developer to get fast feedback on the success and implications of his or her proposed change.  You also get better value from your test environments as a greater ratio of time is spent running tests rather than preparing to run tests.

With more tests run more frequently and earlier in the cycle, the net result is that problems are discovered early and fixed more quickly, as illustrated below.

Virtual databases are a key enabler to Agile Development, allowing IT to produce higher quality releases, faster.  Be Agile not Fragile.