ADMIRAL 20110120 Phase 2 transition
From ImageWeb
Contents |
ADMIRAL Phase 2 transition notes
See also: ADMIRAL_planning_phase_2.
These are a collection of rough notes, some of which overlap other project notes, that are my attempt to summarize where we stand and where we need to direct efforts as move into Phase 2 of the ADMIRAL project. I'm including them here mainly to ensure any thoughts here don't get lost.
Phase 2 started Jan 2011, with primary focus on deployment to research users, in particular to help them get to the point of submitting real datasets.
We held a sprint planning meeting, notes of which are at ADMIRAL_PlanMeeting_16. The sprint review notes are at ADMIRAL_SprintReview_16.
Main goals arising from the meeting included:
- More data storage capacity for the Silk group
- Deployment follow-up with the Development group
- Deploy to additional research groups
- Discuss requirements (possibly for field deployment) with Elephant group
- Improve submission interface in response to researcher feedback
- Improving system support capabilities
- Revise ADMIRAL software deployment structure to make upgrading easier
- Implement script for migrating configuration to an upgraded ADMIRAL server system
- Web tools for user administration
- Improve test suites
Once we can start getting real researcher data submitted, we can address tasks to improve dataset annotations based on researcher feedback.
Silk group and networked storage
For the silk group, we need to increase available disk capacity, since they used up their initial allocation, then stopped using ADMIRAL (without telling us!).
Since they started using ADMIRAL, their requirements have changed and they wish to store up to 4 Terabytes of video data, a storage capacity that exceeds the ADMIRAL project budget's capability to deliver.
ADMIRAL use of departmental storage facility
With very opportune timing, the Zoology department has recently funded the installation of a network storage array facility, to disk storage capacity can be added on an as-needed basis. This facility is to be paid for on a cost recovery basis by projects that use it. We have performed tests to prove that we can use this facility within the ADMIRAL data management framework, and are now waiting for the Silk Group and IT Support to order and install the additional storage capacity so we can deploy an enlarged ADMIRAL server to meet the Silk Group's storage requirements.
On a hardware cost recovery basis, allowing for a normal rate of disk failures over a 5 year period, but not including departmental IT support personnel costs, the estimated cost of providing this facility is about £400 per formatted Terabyte.
The recently released BBSRC data sharing requirements stipulate that research data should be made available for sharing for at least 10 years beyond the life of a research project (i.e. a total of 12-14 years for a typical BSRC-funded project), and claims to allow the cost of this to be added to a research grant proposal to cover the costs thus incurred. In discussions with our IT support team, we estimate that this duration of required availability will approximately double the hardware-only cost of delivering on such a requirement. Based on these discussions, my current recommendation to Zoology department researchers bidding to meet these requirements would be to cost data preservation and sharing to meet BSRC requirements at £1000/Tb, assuming that departmental IT support remains committed to operational management of the storage server. I would also recommend that they allow 1 person day for each month of the project at FEC £400-500 to cover data management support, especially if the research team itself does not have IT server systems expertise: I estimate this would reasonably cover ongoing support of an ADMIRAL-like system for a project.
For comparison, OUCS offers a 5 year data archive facility with multiple offsite tape backups for a Full Economic Cost of about £4000/Terabyte. They do not offer an archive service of longer duration. (It is important to distinguish here between a backup service and an archive service: the same OUCS facility provides a daily backup service, but said backups are destroyed after just a few months of no update or access.)
ADMIRAL use of departmental virtual hosting management
We are also engaging with departmental IT support for virtual machine management of the ADMIRAL systems. We discussed the choice of virtualization platform with IT support in the early stages of ADMIRAL development, and followed their recommendation. We now are realizing an advantage from this, as we are able to include our virtual hosting environment as part of the departmental VMWare management console environment, thus are securing a degree of departmental support for ADMIRAL beyond the life of the present project.
Development group engagement
We deployed a system to the Evolutionary Development group in December 2010. After some initial use to prove the system works for them, they are waiting until they have enough time away from more urgent tasks to review how they'd like to use the facilities provided. (Essentially, to formulate a data management plan?)
Since that deployment, we've made a number of changes to the underlying ADMIRAL platform aimed at facilitating maintenance and migration to new versions. When we have tested these changes in our own instance of ADMIRAL, we intend to roll out an updated version to the Development Group before they get to the point of making heavy use of the platform. One issue we will have when performing software upgrades is that we have not yet devised a way to automatically migrate user account passwords without compromising system security.
Behaviour and Elephant Groups
We aim to engage with these groups just as soon as the current round of ADMIRAL changes have been tested and deployed with our existing research groups.
Current impediments to progress
- Changes to Databank have broken the test suite and dataset submission facilities; we need to have a stable Databank and test suite.
- Deployment of increased storage capacity for the Silk Group
- Further engagement with the Development group: they checked out ADMIRAL, confirmed that it works for them, and have halted further work with it until they have time to think about how to organize their data.
- Engagement with Behaviour Group. We had been holding off pushing this until some system quality and manageability issues were resolved. We are now ready to push ahead with an ADMIRAL deployment for them.
- Engagement with the Elephant Group. We are waiting for a follow-up from our meeting with them last December.
- Fix and upgrade broken MacBook (this is the intended platform for the JISC conference demonstration of ADMIRAL).
