NMNH Information Technology Report: summary of the on-line system and the server crash that forced the cancellation of the RTP '05:

An on-line system was originally developed in 1997 by the National Museum of Natural History (NMNH) using a combination of CGI and Perl programming to help manage application submission and review of the Museum's premiere academic internship program, the Research Training Program (RTP). The automated system was developed out of absolute need. The ever increasing interest in the RTP, resulting in exponential growth in application submission - peeking at 600 applications and over 2,400 document parts - and increased pressure for formal review processes, taxed available staff resources. Through automation, the processes were covered using minimal staff effort with more customers/applicants served and a higher level of quality service provided. The on-line system allowed application documents to be submitted electronically, first through simple e-mail and later via the web. The electronic system was very well received by student applicants, academic references, as well as applicant reviewers and program advisors. The system proved successful by reducing staff time needed to process documents, respond to inquiries, produce reports, and overall improved customer service. The on-line system was revised several times by several different contracting firms and individual contractors using new technologies as the years progressed.

In 2001, following the request of Smithsonian's Office of the Assistant Secretary for Science, the NMNH on-line system was offered to serve as an institution-wide system to administer other internship programs, pan-institution, as well as to accommodate electronic application submission and review for fellowship programs. With base funding provided by Office of the Chief Information Officer (OCIO), Natural History's system was redesigned under the leadership and supervision of the Office of Fellowships (OF), Smithsonian Center for Education and Museum Studies (SCEMS) and OCIO. The new pan-institutional system was named SOLAA (Smithsonian On-Line Academic Appointments). Like the Natural History system, the SOLAA system allowed for complete on-line administration of internship application submission, processing, review and selection. Through SOLAA, after an on-line document was submitted, the submitter could track the status of the document folder including updating the application form and cover letter as needed. Supporting material such as letters of recommendation, resume, transcript, etc. could be added to the document file. A designated program administrator managed the application process by assigning and delegating administrative tasks to others through a variety of options. In addition, SOLAA facilitated an on-line review and selection process, including advanced features for normalization of review scores, score tallying, data sorting, and reporting status back to applicants.

As of February 2005 there were 97 unique programs set up in the SOLAA system supporting the 46 different Smithsonian units who administer internship programs. Of the 46 different Smithsonian units, only NMNH had fully converted all their programs to on-line administration, hosting 33, or 34%, of the on-line units in SOLAA.

The SOLAA system was opened to the public for on-line submission to NMNH programs in the spring of 2002 to accommodate application submission for the 2003 summer RTP program. The SOLAA system was first targeted to be fully deployed by all units in the fall of 2002, in time for 2003 summer programs, but changes in functionality, discovered during parallel testing of 2001 data, deferred deployment. During production, NMNH continued one-to-one parallel testing of SOLAA using 2002 Natural History system on-line submitted data. During this testing the systems functionality was noted, and problems corrected by the developer who accepted a separate contract, paid by NMNH, for specific system fixes. Because of its potential, SOLAA was continuously made available to other Smithsonian units beginning in the spring of 2002 but no other unit elected to fully test or deploy programs through the system until summer 2004. Since the system was first deployed in 2001, there had been some additional development work, mainly focused on correcting diagnostic errors and bringing the system up to parallel functionality of the predecessor Natural History system. Although competitive bids were sought and secured, a single contractor (Diana Specht, a former employee, NMNH Web Master) has consistently provided the lowest bid and therefore served as the main system developer since 1999.

Following preliminary efforts that began in 2002 to convene a review panel to aid in the SOLAA system development and production, in July 2004, the SOLAA Oversight Committee (SOC) was officially established and formally chartered. The role of SOC is to ensure pan-Institutional representation, decision-making, and collaboration among various offices in the management and growth of the SOLAA system. SOC includes representatives from the Internship Council including staff from the Office of Fellowships, Smithsonian Center for Education and Museum Studies, National Museum of Natural History, National Museum of American History, Smithsonian Environmental Research Center, Office of Development, as well as individuals from Office of the Chief Information Officer (OCIO).

At the time of the SOC inception, several members felt that SOLAA had both functional and technical deficiencies that inhibited its use on a pan-Institutional scale, although it worked well for NMNH programs and was very well received by users. Although SOLAA was to remain in full operation and use by any unit electing to automate their programs and processes, in November 2004, OCIO initiated a project under the SOC sponsorship to replace SOLAA, including consideration of using different IT tools and technologies. Several Commercial Off The Shelf (COTS) products are being considered, including Quickhire, as well as development of a system in-house. As of July 2005, nine months later, the project was still in the requirements gathering phase and many months, if not years away from full system deployment and production. Given that many summer 2006 application processes open a year prior, neither Quickhire nor a custom developed system will be available in time to support 2006 intern programs' application cycle.

Building on successes in using the SOLAA system, and in light of furthering automation, especially to better serve NMNH off-site facilities (Ft. Pierce, Anchorage, etc.) through electronic access, during the spring of 2004 NMNH sought to separately redevelop aspects of SOLAA into a new and highly configurable web-based system that would replace a variety of the paper-based processes currently in place at NMNH including academic appointment administration for adjunct scientist, research associate, collaborator, and student; ID badge and proximity authorizations including tracking of volunteers and academic contractors; curator peer review; and on-line administration of small grants and awards including peer recognition, science achievement, etc. The Natural History On-line Academic Administration system (nicknamed NOLAA) was approved for development and implementation by an OCIO team, and funded by the Natural History Director. NOLAA was developed based on the SOLAA foundation code, but uses a higher capacity and TRM compliant database management system - MS SQL. At the request of OCIO to test redevelopment of a ColdFusion (Coldfusion 5.0 to Coldfusion MX) application into new technologies, original reconfiguration of NOLAA was designed to run in a J2EE environment using BEA weblogic. In January 2005 basic programming structure of NOLAA had been completed but several key elements (e.g. security protocols including SSL and data encryption) had not yet been implemented and the units in the system had not yet been fully populated. In January 2005 preliminary testing of the system was completed and found to be very successful in meeting the needs and requirements of Natural History's planned use but further testing was identified as well as security concerns. During NOLAA development, and SOLAA redevelopment, it was assumed that NMNH academic programs would remain open and continue to use the SOLAA on-line system until new systems could be fully deployed.

On 9 February 2005, the Web4 server, the hardware that hosted the SOLAA system and database, experienced a complete hard drive failure when an unprecedented 2 of the 4 hard drives failed at the same time. Attempts to restart the system using the failed disks further complicated the potential to restore any lost data. The age of the out-of-warranty web server and its high utilization by other systems may have contributed to the simultaneous failure of the two hard drives. Many of the websites were hosted on this server because they included legacy systems using older technology (Coldfusion 5.0 versus Coldfusion MX) or non-standard production software (MS Access versus MS SQL databases) and therefore OCIO determined that they could not be effectively hosted in a more reliable server environment.

At the time of the Web4 server crash the SOLAA system was in full use by all NMNH programs and a few other units also had active, although small programs. At the time of the server crash there were approximately 1,000 unique application sets in the system, including archive data from prior years as well as approximately 300 current year application sets in various stages of submission and review. Of the entire SOLAA system data, an estimated 95% of the data was linked to NMNH programs. At the time of the server crash all functions administered by NMNH through the SOLAA system were working well although some other units reported specialized problems.

As a result of the Web4 crash, multiple Smithsonian websites and web-based applications were off-line for several days/weeks, until they could be moved to another web server after restoring them from backup tapes. The SOLAA data lost was further compounded when it was discovered that the directory containing the SOLAA system application and database hosted on the server had been removed from back up protocols. The SOLAA data had never been backed up so no data was recoverable from back up systems. Furthermore, the damaged disks were sent to an outside vendor who was unable to recover any of the missing SOLAA data. With the help of the NOLAA contractor who had used the SOLAA programming and database as a basis for NOLAA development, OCIO was able to locate and restore a previous copy of the SOLAA database, dated 11 December 2004.

The Web4 server crashed during the most critical time of the RTP annual cycle and the restore from December still left more than 95% of the annual submissions to this program irretrievably lost. At least 176 application sets had been received for the RTP and two levels of review completed, involving input scores from two different panels. It's estimated that the other 12 NMNH programs active in SOLAA at the time of the server crash collectively lost between 50 - 100 applicant files, although accurate estimates can not be determined. As a result of unrecoverable lost data from both customers submitting application documents and reviewers evaluating application sets, the imposing timetable, and concerns about fair and equal review, NMNH decided to cancel the 2005 RTP which would have celebrated its Silver Anniversary, its 25th year.

Following the Web4 server crash OCIO made the determination that SOLAA would not be returned to operational status because of security and other technology-related concerns. SOLAA contained sensitive (privacy) data about some applicants, such as social security number. SOLAA data were accessible from the Institution's public website but data protection protocols had not kept up with industry standards resulting in these data not being adequately protected. In addition, appropriate security measures such as encryption of user IDs and passwords had not been implemented and a Privacy Impact Statement, that details what privacy related data is collected, why it's collected, what the Institution does with it, and how we safeguard it had not been posted for SOLAA. Finally, OCIO determined that the SOLAA database required upgrade from MS Access to MS SQL and the programming language from Coldfusion 5.0 to Coldfusion MX.

On 22 March 2005, the CIO and the NMNH Director met to review the OCIO failure to backup the SOLAA database, discuss detailed actions that OCIO will implement to safeguard that such an incident doesn't happen again, and address how best to restore Natural History's on-line capabilities to electronically administer its programs and processes, especially in time to support 2006 programs. As a result of this meeting the CIO agreed to work with NMNH to help take the necessary steps to escalate NOLAA deployment as a quicker and easier replacement for SOLAA.

Because of its configurability NOLAA provided a quick and easy solution for restoring on-line administration of internship programs. Deployment of NOLAA eliminated the need to return SOLAA to public facing production because NMNH internship program on-line functions can be accomplished through deployment of NOLAA. The rationale for this decision was that NOLAA had already been upgraded to TRM-compliant MS SQL and Coldfusion MX and provides additional functionality to support the automation of a variety of NMNH paper-based processes. Furthermore, the OCIO required Life Cycle documentation for public facing systems could be prepared for NOLAA.