ADMIRAL Project Code Structure
From ImageWeb
ADMIRAL Project Code Structure
General Project Code Structure
Code in Google Code repository at http://code.google.com/p/admiral-jiscmrd/.
admiral-jiscmrd |-> admiral-base |-> docs |-> spike |-> src |-> test
admiral-base
Contains code that is used to generate and configure a running ADMIRAL system; most of the code is in the form of shell scripts to perform various tasks.
www
- This directory contains a template of the local web site for each generated ADMIRAL instance. (Selected files are overridden by files in the installation-specific directories). Note that there is also a file admiral-config.js that is a template used to supply installation-specific configuration information for the Javascript parts of the ADMIRAL web applications.
- Other key files include:
- index.html (serves as ADMIRAL homepage, usually overridden by an installation-specific file)
- general css, docs, images and javascript files serving the index file and the overall html files in the source.
Zakynthos (ADMIRAL test machine related)
zoo-admiral-ibrg (IBRG related)
zoo-admiral-silk (Silk Group related)
zoo-admiral-devel (Evolutionay Development group related)
zoo-admiral-behav (Behavioural Ecology group related)
- Contain installation-specific host configuration file hostconfig.sh (has information about the ADMIRAL hostname, workgroup name, datbankhost, DataBanksilo) which helps the other bash and python scripts to retrieve information.
- Dedicated
www/index.htmlfiles for each of the Research groups
Some of the high-level bash scripts
admiraluseradd.sh- Usage: admiraluseradd.sh username fullname role [room number] [work phone] [password]
- This script adds an ADMIRAL user.
admiraluserdel.sh- Usage: admiraluserdel.sh username [purge]
- This script deletes an ADMIRAL user.
- Purge: Including this option also deletes all the user data on the system
- Not using purge orphans the files created by the user being deleted and just deletes the user.
addtestusers.sh- This script adds the test users.
- TestUser1 "Test User 1" RGMember
- TestUser2 "Test User 2" RGMember
- TestLeader "Test Leader" RGLeader
- TestCollab "Test Collaborator" RGCollaborator
- TestUserD “Deleted User” RGMember
- Also essential to run system tests in the area:
- /googlecode-admiral-jiscmrd/test/FileShare/tests
deltestusers.sh- This script deletes the added test users.
- TestUser1 "Test User 1" RGMember
- TestUser2 "Test User 2" RGMember
- TestLeader "Test Leader" RGLeader
- TestCollab "Test Collaborator" RGCollaborator
- TestUserD “Deleted User” RGMember
addorphanedtestuser.sh- This script adds an orphaned user.
- i.e. It deletes TestUserD without purge option orphaning the files and folders created by him on the system.
admiraldataaccess.sh- This script sets access rights (ACLs) on ADMIRAL data file areas.
admiraldatasetup.sh- This script adds and sets up different types of system groups:-
- RGLeader
- RGMember
- RGCollaborator
- RGOrphan
admiralmigrateusers.sh- Generates system configuration for named users.
- Usage: admiralmigrateusers.sh username
- Generates system configuration for all recorded ADMIRAL users.
- Each user gets a password of the form 'username-password' which they should change on first login.
- Usage: admiralmigrateusers.sh all password
- Scans all files for configured ADMIRAL users and sets file ownership and permissions.
- Usage: admiralmigrateusers.sh fileaccess
admiraltoolsetup.sh- Extracts admiral tools from code repository.
admiraluserlist.sh- Lists all ADMIRAL users of the group.
admiralusermanagement.sh- Has useful general purpose management functions.
- Generates user record
- Sets data and owner access (ACLs)
- Generates system user
- Generates system user's home directory
apache-DataBank-proxy- Generates apache proxy settings for the DataBank.
- This file is copied by firstboot.sh and regenerated by createapacheuserconfig.sh
apache-default- Generates apache default settings.
apache-default-ssl- Generates apache default ssl settings.
config-files- Provides a listing of all the ADMIRAL config files required to generate a working ADMIRAL system.
createapacheuserconfig.sh- Generates Apache access configuration for named user (or deleted user)
- Usage: createapacheuserconfig.sh username [orphan]
- Generates Apache access configuration for all configured ADMIRAL users
- Usage: createapacheuserconfig.sh all
firstboot.sh- This script will run the first time the virtual machine boots and is run as root.
lvmconfig.sh- Contains commands to create a data volume on the disk partition at /dev/sdb1.
- Copies files from the /home/data directory, and symlinks /home/data to the new data location.
- This is the default case for a new system build when using a single separate disk volume for ADMIRAL data.
make-admiral-server.sh- Builds the ADMIRAL server.
makeresearchgroupfiles.sh- USAGE: makeresearchgroupfiles.sh(test|copy) hostname password [targetdir]
- Copies files from source directory to target directory, ignoring those listed as blacklisted, and replaces the originals with a symlink to the copy.
- This is run prior to running updateadmiralfiles.sh
updateadmiralfiles.sh- USAGE: updateadmiralfiles.sh RUN
- This updates the ADMIRAL configuration files on system.
- Before running this, makeresearchgroupfiles.sh needs to be run to create a customized copy of the installation build files.
postboot-1.sh, postboot-2.sh, postboot-3.sh, postboot-4.sh- These are the Postboot scripts.
- More information is available in the build documentation available in the ADMIRAL Project wiki.
docs
Contains clipart directory and other images-png, pdf and svg files that have been used to create site images.
spike
This directory serves as an experimental spike area that was created to test the functionality and working before incorporating it into the real code. ajaxcode – Contains some useful code for DataBank listing using ajax applications → Admiral → RESTful – Contains some useful code for RESTful applications. Likewise most of the the ADMIRAL submission interface spike can be found in this area. cursample rdf query spike others useful code
src
src
|-> AdminUI
|-> AdminUIhandler
|-> Admiral
|-> AllTests
|-> DatasetUtils
|-> DisplayDataset
|-> images
|-> jQuery
|-> MiscLib
|-> MochiKit
|-> qunit
|-> rdfquery
|-> SelectDataset
|-> SubmitDatasetHandler
|-> SubmitDatasetUI
Admiral
Contains some of the base level files used in most of the user interfaces.
admiral-base.js- Contains some of the frequently used base level functions to simplify the code further.
Async-Computation.js- Contains class for controlling and composing asynchronous computations, inspired by the monad concept used by Haskell and other functional programminglanguages.
tests- This contains all the tests wriiten for the AdminUI.
- Other files and folders
- Contains MiscLib, Utils and session related python files.
AllTests
AllTests.html- This runs all the java script tests of the submission interface.
AllTests.py- This runs all the server side python tests of the submission tool.
DatastUtils
Contains some of the common code written for DataBank-dataset related operations.
DisplayDataset
DisplayDataset.html- This page displays the dataset information obtained from the DataBank
- This serves as a customised ADMIRAL view of the DataBank dataset.
- The folder contains the related css, js and tests.
Images
The folder serves as an ADMIRAL image repository used in the tool.
jQuery
The most extensively used javascript files can be found in this directory. Query tree view JS files can be seen here.
MiscLib
The miscellaneous js files used can be found here.
MochiKit
The MochiKit related js files can be found here.
qunit
The js qunit test related files can be found here.
rdfquery
The RDF realted jquery files can be found here
SelectDataset
Web page and code to enumerate available datasets stored in Databank, and select one for detailed display.
SelectDataset.html- Helps in enlisting all the datasets from a particular silo in the DataBank submitted by all the members of a research group.
GetDatasetDetails.js- Obtains the dataset related details of a supplied dataset identifier.
GetDatasetsList.js- Obtains all the datasets from a particular silo in the DataBank submitted by all the members of a research group.
ListDatasets.js- Handles the UI js to display all the datasets returned from GetDatasetsList.js
SelectDatasetHandler
The server side code of the submission interface is a cgi application written in python.
DatasetDetailsErrorHandler.py- Main CGI program that handles error page after dataset selection before submission to RDF DataBank.
DirectoryContentsListingHandler.py- Directory Contents Listing handler lists all the directory contents both files and sub-directories from ADMIRAL to the requesting module.
DirectoryListingHandler.py- Directory Listing handler lists all directories from ADMIRAL to the requesting module.
GetDatasetMetadataHandler.py- Get Data submission Handler helps in populating metadata for the requested dataset directory.
SubmitDatasetConfirmationHandler.py- Main CGI handler program for submitting data files from ADMIRAL to be stored as an RDF DataBank dataset.
SubmitDatasetDetailsHandler.py- Main CGI handler program for saving metadata that handles the dataset submission details page.
SubmitDatasetSummaryHandler.py- Main CGI program that handles response summary page after dataset submission to RDF DataBank.
HttpSession.py- Low-level HTTP session functions to support DataBank RESTful API operations.
ManifestRDFUtils.py- Support functions for creating, reading, writing and updating manifest RDF file.
SubmitDatasetUtils.py- Support functions for creating, unpacking and managing datasets in RDF DataBank.
MiscLib- Miscellaneous library functions used in the submission interface tool.
tests- Contains all the python tests that test each of the functionality used in the tool.
- Tests were written as part of the test-driven-development approach.
- These tests can also be run from the Alltests directory mentioned previously.
SelectDatasetUI
The two pages forming the Submission UI Interface are listed below.
SubmitDatasetdetails.html- Submisssion UI Details page
SubmitDatasetConfirmation.html- Submission UI Confirmation page
AllContentsDatasetTree.js- Converts the file segment tree structure into a jQuery nested directory list structure.
DirectoriesOnlyDatasetTree.js- Converts a file segments tree structure into a jQuery nested directory list structure.
DirectoryContentsListing.js- Makes a call to the ADMIRAL directory contents listing handler to provide a listing of the directory contents.
DirectoryListing.js- Makes a call to the ADMIRAL directory listing handler to provide a directory listing for dataset selection.
GetDatasetMetadata.js- Gets the form field information from the ADMIRAL metadata.
- Makes a call to the ADMIRAL metadata handler.
SubmitDatasetConfirmation.js- Populates the form fields from the ADMIRAL metadata and submits the dataset to the DataBank.
SubmitDatasetDetails.js- Reads from the admiral manifest if any exists and populates form fields and provides a directory listing for selection.
URIUtils.js- Converts the URI with unicode to a displayable string.
test- Contains all the javascript tests that tests the functionality used in the UI interface. Tests were written as part of the test-driven-development approach.
- These tests can also be run from the Alltests directory mentioned previously.
AdminUI
User administration is designed around a RESTful API and a small number of web pages that provide a user interface for driving the functions thus provided.
The two pages forming the User administration UI Interface are listed below.
AdminFrontPage.html – Admin UI Front pageUserDetailsPage.html – Admin UI Details page- This directory contains the Admin interface related *: UI files.
- It contains css, html , js MiscLib files.
- Also additionally, the RESTful APIs along with a readme is enlisted to assist further development of the interface.
- This feature is yet to be completed.
ADMIRAL Admin Interface API
-
- http://hostname/users
- http://hostname/user/user_id
- Users
- /users/{user_id}
- GET:
- Action: list users
- Returns json listing all the ’ADMIRAL users’
- Returns: 200 for success, 401 Unauthorized
- POST:
- Action: add user
- Creates an ADMIRAL user with the supplied user_id and details
- Details: Full Name, Role (user/collaborator/group leader), Room Number, Phone, Password
- Returns: 201/204 for success, 400 for bad request, 401 Unauthorized
- User
- /user/{user_id}
- GET:
- Action: user details
- Returns all the user details in json format for the specified ADMIRAL user_id.
- Details: Full Name, Role (user/collaborator/group leader), Room Number, Phone, Password
- Returns: 200 for success, 401 Unauthorized
- PUT:
- Action: modify user details
- Modifies the user details for the specified ADMIRAL user_id
- Details: Full Name, Role (user/collaborator/group leader), Room Number, Phone, Password
- Returns: Â 200 on update success, 400 for bad request, 401 Unauthorized
- DELETE:
- Action: delete user
- Deletes the specified user with user_id from the ADMIRAL system
- Returns: 200 on successful delete, 404 if user with user_id doesn’t exist, 401 Unauthorized
-
Installing and starting the user administration services
- 1> Install web.py : http://webpy.org/install
- 2> Install smbldap-tools using apt-get
- 3> Run the RESTful application:
- $ python \googlecode-admiral-jiscmrd\src\AdminUIHandler\services\AdminUIHandler.py
- http://0.0.0.0:8080/
- 4> (proxy... modwsgi replaces proxy, may be)
- ...
- 5> Access the Admin UI interface from the browser
- The following lists all the Admiral users in json format:
- ex: http://zoo-admiral-ibrg.zoo.ox.ac.uk/users
- The following lists the Admiral user details in json format:
- ex: http://zoo-admiral-ibrg.zoo.ox.ac.uk/user/TestUser1
- Note:
- Default port for webpy :8080
- \googlecode-admiral-jiscmrd\admiral-base - Contains scripts (.sh files)
- \googlecode-admiral-jiscmrd\src\AdminUI - Contains UI (html, css, js)
- \googlecode-admiral-jiscmrd\src\AdminUIHandler - Contains RESTful services ( .py ) ....
- The following .sh files are added to admiral-base to help the Admin UI Interface
- 1> listAdmiralUsers.sh
- # $1 RemoteUserID ex: TestUser1
- # $2 FolderPath ex: /home/data/private
- 2> admiraluserinfo.sh
- # $1 RemoteUserID requesting the UserID details
- # $2 UserID whose details are being requested by RemoteUserID
- 3> admiralupdateuserinfo.sh
- # $1 RemoteUserID trying to update the UserID details
- # $2 UserID for whose details are being updated
- # $3 FullName Updated FullName of the Admiral User with ID=UserID
- # $4 Role Updated Role of the Admiral User with ID=UserID
- # $5 RoomNumber Updated Room Number of the Admiral User with ID=UserID
- # $6 WorkPhone Updated Work Phone Number of the Admiral User with ID=UserID
- # $7 Password Updated Password for the Admiral User with ID=UserID
- To Do : ( look at authentication issues for remote user ID)
AdminUIhandler
- Services
- AdminUIhandler.py
- This serves as the server side handler for the Admin UI interface requests.
- tests
- This contains all the tests wriiten for the AdminUI.
- Other files and folders
- Contains MiscLib, Utils and session related python files.
test
test
|-> FileShare
|-> MiscLib
|-> RDFDataBank
|-> TestLib
|-> Ziptest
FileShare
This contains all the tests that test the working and access of the private/shared/collab areas by RGLeader, RGMembers and RGCollaborators.
MiscLib
This contains miscellaneous python files and utility functions that help write the actual code.
RDFDataBank
Contains RDFDataBank related tests obtained from the DataBank github area.
TestLib
Contains Sparql related tests.
ZipTest
Contains tests that test the working of the dataset zipping to be sent to the DataBank.
