This week is an exciting week for the Indaba fieldwork platform. After several months of design, development, and testing, we went live today with some new and powerful features, notably Control Panel (which allows project managers to more easily design and launch their projects) and important enhancements to Fieldwork Manager (like being able to send Question Lists to any user on a project at any point in the workflow) We'll dive more deeply into these upgrades and what they mean for running more powerful and efficient Indaba projects in a separate post.

Less visible is the fact that we've also quietly launched an experimental approach to importing legacy data into Indaba from other systems/data files. This has been a use case identified for many years by Indaba users (including us here at Global Integrity): "Great that I can run my new project on Indaba, but is there a way for me to migrate the past ten years of data I've already published into the platform?" Until now the answer was basically, "No," absent a willingness to brute force the solution by copying and pasting potentially thousands of spreadsheet cells or published web pages into a "new" Indaba project. And frankly no one wants to do that.

To try and solve this challenge, we're partnering with Seabourne and longtime Indaba developers OpenConcept Systems (OCS) to test a machine-driven method for importing legacy data into Indaba. Here's the basic approach:

– We've developed a template and data schema for what is necessary to import questionnaire-style information into Indaba in a structured manner that meets Indaba's data model needs.

– Seaborne's nifty Delray software — which "eats" random unstructured data from disparate sources and then spits it back out in a fully customized and structured package — will then eat several years' worth of Global Integrity legacy data currently hosted off of Indaba. In practice we are talking about the Global Integrity Reports from 2006 – 2009 and a few early Local Integrity Initiative projects.

– Delray will then give us a nice, Indaba-ready set of files containing our old data that is ready for import.

– Our friends at OCS will then take the import file, drop it into Indaba's core database, and (hopefully!) our questionnaires will automagically rebuild themselves without any human intervention. We'll then be able to use Publisher's standard features, such as web widgets, to quickly republish hundreds of pages of legacy reports and scorecards on our website.

Naturally, something probably will and should go wrong the first time we try this. But we're hoping to test this process on ourselves first, and if/when it works be able to offer it to others in the future at low cost to help with legacy data migration needs. Stay tuned for more details in the coming weeks.

— Nathaniel Heller