- What is a data warehouse?
A data warehouse is a repository of data from many sources. Unlike a typical "transactional" system used for day-to-day operations, a data warehouse is structured to maintain large amounts of related, historical data for analysis and reporting. A data warehouse provides for easy reconstruction of "snapshots" of historical data, as well as the ability to link such snapshots over time using certain criteria.
- Why do we need a data warehouse?
Improving educational performance and accountability depends on understanding the relationships among areas such as curricula, assessments, special programs, teacher qualifications, program spending, discipline incidents and attendance. With our current systems it is difficult to cross-reference student-to-teacher, student-to-course, or student-to-program information. Without the ability to link data, Department and district decision-makers cannot leverage the full potential of the information we collect and process.
The data warehouse is a "longitudinal data system" capable of linking student, teacher and financial information over multiple years, across multiple schools and districts. Using the warehouse, Department and district decision-makers can take key metrics from multiple areas and analyze them in a single view. For example:
- What are the results of program participation in terms of achievement or graduation rates?
- Which programs targeted at which student population have the greatest impact?
In addition to supporting data-driven decision-making, the data warehouse provides districts with timely access to student SIMS and MCAS history, which is especially important for supporting students transferring across districts.
- Who is developing the data warehouse for the state?
The Department worked with IBM/Cognos Corporation, an organization that specializes in large-scale data management systems. Cognos was selected through a competitive procurement process. Similar Cognos educational warehouse systems are operational today in 18 departments of education worldwide.
A state developed and maintained data warehouse saves districts these expenses while still enabling them to populate the warehouse with historical data from their own administrative systems. The accuracy and consistency of the district's warehoused data will depend on the accuracy and consistency of the data in a district's administrative systems.
- Will this system replace a district's or a school's current student information system (SIS)?
No. The data warehouse is capable of drawing data from many sources, including local SIS applications. It is designed to easily load archive-ready data "in bulk" at regular intervals and to provide enhanced analysis and reporting on interrelated data from multiple sources and time periods. The data warehouse is not meant to manage current day-to-day transactions.
Many SIS vendors have been working with the EDW team to create export scripts that will facilitate loading local data into the data warehouse. Contact your SIS vendor to find out what the status is on the development of these scripts and export tools.
- Where will the data in the warehouse come from?
The data in the warehouse comes from the state and local sources. The Department has loaded MCAS/MEPA and SIMS data from 2002 to the present into the warehouse. Districts can load local data, including student, course, staff, grades, GPA, and local assessments. For information about loading local data, contact the Data Warehouse staff at datawarehouse@doe.mass.edu.
In the near future, the ESE will load October 2007 EPIMS data. The warehouse may also be expanded to include discipline and attendance detail, financial, and program data. The primary source of local warehoused data will be school/district administrative systems, including HR, SIS, grading, and assessment systems. Districts can periodically upload data to the warehouse as needed. The data warehouse will provide districts with the ability to maintain multiple years of integrated, historical data for analysis and reporting.
- Who will have access to my district's warehoused data?
Local data uploaded by a district cannot be viewed by other districts and is accessible only to those district staff assigned data warehouse roles in Directory Administration. The state's access to local data is limited to such rare cases as the required review of a district's files, in which case the district would be notified prior to such a review.
All users can access SIMS and MCAS data in aggregated form, as with public profile reports; but only those district users assigned the authorized roles in Directory Administration will have access to more confidential data pertaining to the students in their district. At the state level, only those Department of Elementary and Secondary Education staff that have a need for the data to fulfill their professional responsibilities will be given access to confidential student SIMS and MCAS data, in line with current Department policies.
For additional information on access to confidential student information, see the Access to SIMS category under Reporting Guidelines Frequently Asked Questions and the Data Warehouse Data Access Policy.
- What tools will districts use to analyze and report on the historical data in the warehouse?
Authorized users may access the warehouse analysis and reporting tools online through the Security Portal. The primary reporting tools are Cognos Connection, the main portal and report viewer; Cognos PowerPlay, a cube-based analysis and reporting tool; and Cognos Report Studio, a report-authoring tool. The Department will provide pre-defined reports and analysis cubes and district staff will have the ability to create additional reports using Report Studio.
- What training will my staff need, and what training resources are available?
The Department has currently identified three general levels of training:
End Users: The Department has contracted with the Public Consulting Group (PCG) to develop six courses on accessing and using the Education Data Warehouse. A trial run of these courses is being presented to current data warehouse users this fall and the final versions of the courses, complete with training materials, will be made available to districts upon completion.
Report Authors: District and Department users will have the ability to create their own reports using Cognos Report Studio. It is recommended that users of Report Studio participate in the trainings being developed by PCG prior to being granted access to the Report Authoring role in Directory Administration.
Data Loading: For districts that opt to load local data, district technical staff responsible for producing data extracts will need to become familiar with the extract formats and procedures for generating and uploading local data extracts. The Department will provide support and training as needed on a district-by-district basis, and will work with district technical staff and SIS vendors to provide the information needed to load the necessary extracts. However, districts should expect to devote considerable time and resources to producing the initial loads. Districts that have loaded local data thus far report that the time spent from extract start to first successful load ranged from three to six months.
- My district is considering incorporating a new assessment into our instructional plan. Should I wait to see if it is supported by the data warehousing system before I decide to use it?
In general, no. The data warehousing system is a tool intended to support policymaking and instruction. When deciding whether to adopt a new assessment, the primary concern should be whether or not it will improve educational outcomes for students, not whether or not it will be compatible with the tool.
With that in mind, we do recommend that schools and districts consider assessments with the following characteristics:
- Can produce results in electronic format. Assessment data must generally be exportable into flat files in order to be loaded. Paper-only results may create cumbersome manual-entry tasks for district staff.
- Can incorporate the use of state identifiers (SASID, MEPID, district/school codes) into result files. This will greatly simplify the task of linking assessment results to existing data.
- Can my district sign up to be a participating district?
Yes. All districts now have access to the Department's Education Data Warehouse. School and district level security roles are available to be assigned to individuals through Directory Administration. It is recommended that districts review the documents available on the Education Data Warehouse homepage, which not only address how to access the data warehouse but the steps districts should take to develop a strategy for the access and use of data. In addition, it is requested that districts notify the data warehouse team prior to loading their local data.
- Should my district consider acquiring its own data warehousing system?
While we cannot prevent districts from doing so, the Department strongly recommends that districts consider the long-term benefits of a statewide data warehousing system. Most district-level systems require an annual per-student fee that can result in heavy long-term costs.
The statewide system, on the other hand, expects to provide a base level of functionality at minimal cost to districts. Extended functionality, including loading of local data, involves upfront costs devoted to building district capacity and training, and complete system ownership after an initial development period — in other words, the long-term costs are expected to be far lower due to district self-sufficiency.
In addition, the Department expects districts to benefit greatly from using a standardized system. Specific reports, professional development, and best practices can be shared easily among districts using a common platform.
- Where can I get more information on the data warehouse?
For additional information on the data warehouse project, check for updates on the public web page or email questions to the data warehouse project team.
State Government · State Services 
