Sample Projects and Reports

SELECTED GRG PROJECTS AND REPORTS

From the over 400 program evaluation and research projects we’ve conducted since 1989, we’ve selected a fairly representative sample of projects within a variety of categories. Please note that a number of projects described below could be placed into multiple categories, since the categories are not mutually exclusive. However, we have selected one category per project for this list (which is presented in alphabetical order).

We have also provided links to project reports and/or executive summaries for some of the projects described here. To see a report associated with a particular project, click on the hyperlink to take you to a PDF.

• • • • • • • Arts and Culture
            • Broadening Participation in Science, Technology, Engineering, and Math (STEM)
            • Climate Change
            • Curricula
            • Early Care and Education
            • Energy and Environment
            • Fellowships and Scholarships
            • Health and Human Development
            • Literacy
            • Media, including Children’s television
            • Museum and Library Programs
            • Partnerships and Collaborations
            • Professional Development
            • Public Engagement with Science
            • Strategic Planning and Evaluation Capacity Building
            • Youth Programming

ARTS AND CULTURE

Metropolitan Opera, New York, NY

GRG evaluated the 2018-19 season of the Metropolitan Opera’s HD Live in Schools program, an initiative to bring the Met’s live HD performances to high school students across the country. GRG had evaluated four prior seasons of HD Live in Schools, focusing on experiences and outcomes for both teachers and students. During the 2018-19 season, GRG surveyed over 1,000 students in grades 4-12 and 45 teachers across the country about their experiences. Participating students learned about opera and its historical, political, and cultural relevance; greatly enjoyed the opera broadcasts; and were more likely to find opera relatable, understandable, and interesting after participating in HD Live in Schools. Teachers were impressed by the educational materials and professional development they received and reported that their students had learned from and enjoyed the program.

Isabella Stewart Gardner Museum, Boston, MA

GRG conducted formative and summative evaluation of the Institute of Museum and Library Services (IMLS)-funded project to create a series of 12 short videos looking at “behind-the-scenes” activities at the museum. The videos were available via on-site kiosks and on the museum’s web site. After the first three videos were made, GRG gathered feedback from a range of individuals to inform development of the remaining films. At the end of the project, we conducted an online survey to assess the videos’ influence on people’s understanding of and appreciation for what goes on behind the scenes in the museum, and how this impacts their personal connection with the museum.

Wadsworth Athenaeum, Hartford, CT

GRG conducted formative evaluation of the museum’s newly redesigned Cabinet of Art and Curiosity exhibit, The exhibit introduces visitors to a unique concept of collecting that flourished in 16th- and 17th-century Europe, and encourages visitors, through traditional and digital interpretive materials, to make connections to the present time. GRG conducted “Talk-along” interviews and intercept surveys with exhibit visitors, with the aim of assessing their experience with the exhibit, as well as how well they comprehended the themes and concepts being presented.

Mannes Productions, in collaboration with WNET-TV, New York  

GRG conducted a formative evaluation of The Music Instinct, an NSF-funded 2-hour PBS television program that aimed to convey the strong evidence of the connections between music and science as well as a deeper understanding of these two fields. The overall project consisted of the PBS program, a web site, and ancillary educational materials. The formative evaluation obtained timely information to support and guide producers as they made decisions regarding the design, content, and format of project components. Pre-production evaluation activities included an online survey of PBS viewers and music aficionados that explores the likelihood of watching such a show. Production evaluation activities included four focus groups of target audience members, an online survey of formal and informal educators and PBS viewers about the website, and a written survey of educators (middle school, after-school, and museum) about the ancillary materials, which included a tool kit and a museum guide.

Harvard Museums of Science and Culture, Cambridge, MA

HMSC are comprised of the Collection of Historical Scientific Instruments, the Harvard Museum of Natural History, the Harvard Semitic Museum, and the Peabody Museum of Archaeology & Ethnology. GRG’s study for HMSC explored and evaluated the overall visitor experience, making recommendations regarding visitor engagement and for purposes of PR and marketing. GRG conducted an exit/intercept survey of over 700 visitors across the four museums and a survey of over 750 attendees at eight spring 2015 HMSC public programs. Our audience research was a strong first step in HMSC improving its understanding of who is visiting, why they are visiting, and what their experiences and outcomes are.  We were able to respond to particular interests of HMSC by comparing results of local residents versus tourists.

Pew Charitable Trusts, Philadelphia, PA

GRG carried out a summative evaluation of EGG the arts show, a weekly TV series produced by WNET NY, intended to increase viewers’ awareness of and participation in the arts. The evaluation was conducted in four cities and included surveys to viewers, focus groups, and interviews. The evaluation’s main goal was to assess the impact of the series on viewers’ global attitudes about the arts, specific interest in the art and artists featured on the show, and arts-related behaviors, including participation in the arts and financial contributions to arts and cultural organizations.

 
BROADENING PARTICIPATION IN STEM

EDC, Waltham, MA

EDC is collaborating with the Connecticut Science Center and the School District of New Britain on Supporting Science Inquiry, Interest, and STEM Thinking for Young Dual Language Learners (SISTEM). The 4-year NSF ITEST-funded project aims to build knowledge about increasing access to the STEM pipeline for prekindergarten Dual Language Learners through family, teacher, and community supports. The project team is using an iterative research design to develop and refine SISTEM with input from families, teachers and content experts, and is conducting quasi-experimental research to assess how participation in SISTEM affects parents, teachers and DLL PreK children. GRG’s evaluation is designed to complement the project’s research by focusing on community uptake of SISTEM (i.e., by school and district PreK leaders) and by summatively evaluating SISTEM’s contributions to ITEST’s five key areas: Innovative use of technologies; Innovative learning experiences; STEM workforce development; Strategies for broadening participation; and Strategic partnerships. Evidence is being collected through key informant interviews and document review.

Indiana University-Purdue University Indianapolis (IUPUI), IN

The Indiana STEM LSAMP (IN LSAMP) consists of three research institutions: Indiana University-Purdue University Indianapolis (IUPUI), Indiana University Bloomington (IUB), Ball State University (BSU); two four-year regional universities: Indiana University Northwest (IUN) and Indiana University South Bend (IUSB); and Ivy Tech Community College (ITCC) Indianapolis,  all working together to increase the participation and advancement of underrepresented students in STEM disciplines. GRG is conducting external formative and summative evaluation. The multi-method design includes online surveys, social network analyses, in-person observations and interviews, and phone interviews with project participants (i.e., students, faculty, staff, and project team across the Alliance institutions).

Girls Get Connected Collaborative, Somerville, MA

GRG completed a 3-year formative and summative evaluation for the Girls Get Connected Collaborative (GGCC) of their Technology at the Crossroads project. With funding from NSF ITEST, GGCC partnered with Patriots’ Trail Girl Scout Council (PTGS), the Boston Public Schools TechBoston program, Simmons College, Boston College’s Urban Ecology Institute (UEI), and other local programs to design/adapt, implement, and disseminate a comprehensive set of STEM curriculum modules that sought to give middle and high school students firsthand experiences in using tools to solve real-world problems, while providing them with the necessary knowledge and skills to pursue further study in these fields.

The formative evaluation informed the development of the curriculum in year one, after which the summative evaluation assessed the influence of the program on participating students and mentors and examined the extent to which the program was implemented as planned. Using authentic assessments, GRG collected data over the three years from middle school youth, high school mentors, undergrad teacher/mentors, teacher leaders, and key players at participating organizations.

The SPLICE (Security and Privacy in the Lifecycle of IoT for Consumer Environments) project is funded by an NSF Frontiers grant to investigators at Dartmouth, Morgan State, the University of Illinois, the University of Michigan, Johns Hopkins, Tufts, and the University of Maryland. GRG is evaluating the activities of SPLICE’s Broadening Participation Committee, which aims to expand the diversity of populations engaged in computing by creating diverse learning communities to encourage understanding the implications of Smart Homes and computing more generally, providing opportunities and support for students from groups underrepresented in computing to participate in research experiences related to Smart Homes and Internet of Things, and creating and implementing educational, training and professional activities to promote the inclusion and development of diverse populations in computing.

Dartmouth College, Hanover, NH

Rural Gateways was designed to enhance rural librarians’ self-efficacy and professional self-identification as adult science program developers through professional development, an online community of practice, and scaffolded experiences of providing adult STEM programming in their communities, with the eventual goal of building their capacity to sustain this programming beyond the life of the project. Dartmouth College collaborated on this NSF-funded grant with Dawson Media Group, Oregon State University, the Association of Rural and Small Libraries, and the Califa Library Group. GRG’s summative evaluation found that librarians made significant gains in self-efficacy, self-identification as science program developers, and preparation for informal science learning leadership, and self-efficacy gains in particular tended to persist. Audiences and science partners also showed positive effects of participation.

 EDC, Waltham, MA

GRG served as external summative evaluator of the Girls Communicating Career Connections (GC3), a project by EDC, funded by NSF Division of Gender in Science and Engineering Program. The GC3 project’s media series–short video segments produced by middle school aged girls– captured the inquiry-based learning experiences of girls as they investigated what it means to be a scientist or engineer. The project materials also included a companion Educator Web site and supporting materials for formal and informal educators. The summative evaluation surveyed both GC3 Educators and GC3 Youth.

 CLIMATE CHANGE

Columbia University, New York, NY

The PoLAR Learning And Responding: PoLAR Climate Partnership, led by Columbia University, engaged adult learners through novel polar educational approaches to advance their understanding of climate change and to stimulate engagement in meaningful individual and collective responses. GRG served as external evaluator of this Phase II project, which was one of six projects in the NSF Climate Change Education Partnership (CCEP) program. We also conducted evaluation during their Phase I planning grant. Throughout Phase II, GRG conducted formative and summative evaluation as the partnership evolved, and as the cohesive suite of educational tools and technology were developed and disseminated. Formatively, we assessed the likely effectiveness of the products and provided suggestions for modification. Summatively, we assessed each product’s effectiveness and the overall project success regarding the communication efforts and intended audience impacts.

Franklin Institute Science Museum, Philadelphia, PA

Funded by the National Science Foundation, The Franklin Institute (TFI) partnered with five organizations to develop a Climate Change Education Partnership (CCEP) project for urban centers in four major northeast cities (Philadelphia, Pittsburgh, New York, and Washington, D.C.). CUSP partners included The University of Pittsburgh Center for Learning in Out of School Environments (UPCLOSE), The Center for Climate Systems Research (a unit of the Earth Institute at Columbia University in New York), Carnegie Museum of Natural History in Pittsburgh, New York Hall of Science, and National Geographic in Washington, DC. GRG evaluated both the Phase I grant (Urban Climate Education Partnership), and the five-year Phase II grant. The CUSP program aimed to convene comprehensive networks of community organizations in the four cities to educate urban audiences about climate science and the impacts on urban environments. GRG evaluated the core CUSP partnership, including aspects of communication, collaboration, leadership and decision-making strategies, and documented and assessed the activities of the local ULN members and corresponding impact on the urban communities.

Brown University, Providence, RI

GRG conducted formative and summative evaluation of the Seasons of Change: Global Warming in Your Backyard for The Watson Institute at Brown University. Seasons of Change, funded by NSF, was a traveling interactive exhibit designed to increase visitors’ awareness of the impacts of global climate change through the use of local harbingers. Evaluation sites included two of the three museums that hosted the exhibit:  the EcoTarium in Worcester, MA and the McAuliffe-Shepard Discovery Center in Concord, NH. The exhibit moved on to North Carolina, with adaptations to the exhibit based on that region’s climate issues.

Clean Air-Cool Planet, New Canaan, CT

GRG conducted both formative and summative evaluation of Connecticut Clean Air, a two-year collaboration between Clean Air-Cool Planet and the Connecticut Clean Energy Fund to develop educational programs and tools to help the public learn about clean energy as a solution to climate change. The evaluation aimed to test and evaluate existing exhibits and curricula to identify materials that would be appropriate or adaptable for a variety of venues in Connecticut. Subsequently, GRG conducted similar work in Massachusetts to evaluate the Clean Energy Climate Solutions Project. This project involved the use of educational kits to educate science center visitors and classroom teachers about clean energy as a solution to climate change. In the evaluation, we tested existing kits and curricula to identify materials that would be appropriate or adaptable for a variety of venues in Massachusetts.

National Geographic Society, Pittsburgh, PA

GRG provided formative evaluation consultation as part of Pittsburgh Climate Change, a planning grant that National Geographic received from the Richard King Mellon Foundation. National Geographic and a consortium of informal and formal education institutions in the Pittsburgh area came together to plan how they could best educate Pittsburgh students about climate change. The partners were seeking to leverage their various resources in order to mount a project with three large components involving professional development for teachers, programs for middle school students, and family/public outreach. Consortium members were: Carnegie Museum of Natural History, Carnegie Science Center, Phipps Conservancy, Western Pennsylvania Conservancy, Pittsburgh Public Schools, Pittsburgh Zoo, Pittsburgh Parks Conservancy, National Aviary, Allegany County Schools, Carnegie Mellon University, and University of Pittsburgh.

CURRICULA

EDC, Waltham, MA

GRG is evaluating EDC’s Young Mathematicians in Worcester Program. Funded by Overdeck Family Foundation and Heising-Simons Foundation, EDC is connecting organizations across Worcester to create a Math Learning Community to improve children’s mathematics knowledge and skills and support families’ positive attitudes toward mathematics. The program includes a 12-hour, 6-session professional development course for educators; mathematics games that can be used in school, at home, in libraries, and playgroups; mathematics minibooks that complement game play; and a resource guide for adults including suggestions for mathematics picture books. GRG’s evaluation includes formative and summative components, providing the project team and funders with an independent perspective and actionable evidence of progress and results.

SAE International (Formerly the Society of Automotive Engineers), Warrendale, PA

For SAE, GRG conducted a five-year longitudinal outcome evaluation of A World in Motion (AWIM), an interdisciplinary engineering supplemental curriculum kit for use in middle and high school grades that was designed to promote science, engineering, and mathematics literacy among students by engaging them in challenging engineering and design activities. Primary outcomes assessed were changes in students’ and teachers’ attitudes, interest, and knowledge.

Tanenbaum Center for Interreligious Understanding, New York, NY

GRG conducted an evaluation of Building Blocks for Democracy, a K-6 curriculum designed to increase interreligious understanding as part of a broader multicultural education approach. GRG’s evaluation identified the full range of current curriculum implementation practices, the features and characteristics of successful implementations, and outcomes for teachers and students.

USC Shoah Foundation Institute for Visual History and Education, Los Angeles, CA

GRG conducted a multi-year evaluation of the multi-media curriculum, Echoes and Reflections, developed through a partnership of the Anti-Defamation League, the Shoah Foundation, and Yad Vashem, The Holocaust Martyrs’ and Heroes’ Remembrance Authority in Jerusalem. The curriculum introduces American teachers to new methods for using primary sources in middle and high school classrooms when teaching about the Holocaust. The multi-phase evaluation: (1) assisted the project team as they finalized the curriculum, (2) documented the two-tiered training, distribution, and implementation processes, and (3) assessed the overall effectiveness of the curriculum in classrooms.

Oregon Public Broadcasting, Portland, OR

GRG conducted formative evaluation of the American Literature curriculum, funded by Annenberg-CPB. The curriculum was comprised of 16 half-hour video episodes, each of which focused on one key movement in American Literature. Instructors at the secondary and post-secondary level completed in-depth surveys that enabled us to provide feedback to the producers regarding the design, production, and implementation of the course’s three main components: the videos, a website (including a searchable database, teacher materials, and student materials), and a printed version of a Teacher’s guide.

Mathematical Association of America, Washington, DC

GRG conducted two years of formative evaluation of a mathematics curriculum project entitled Historical Modules for the Teaching and Learning of Secondary Mathematics. During the first year, GRG field-tested and evaluated six curriculum modules for the project. The goal of this NSF-funded project was to enhance the learning experience of middle and high school students through showing mathematics as a subject rich in history, people, ideas, and tradition. The findings from our formative evaluation helped the curriculum developers make critical revisions to the modules. During the second year, GRG carried out further evaluation, this time exploring the effectiveness of a second set of six curriculum modules.

Bronx Zoo, Wildlife Conservation Society, Bronx, NY

GRG carried out a 2½-year evaluation of the Bronx Zoo’s zoo education Kgn-3^rd grade curriculum, Pablo Python Looks at Animals.  The research project used a quasi-experimental design with pre- and post-testing of several hundred students on their understanding of content and mastery of the concepts presented in the curriculum.  GRG also assessed teachers’ comfort in the life science classroom and their knowledge of the subject matter they were teaching.

TERC, Cambridge, MA

GRG conducted five separate evaluations for TERC of their curricula and research projects.  One was a summative evaluation of Study of Place, two web-delivered curriculum modules, each framed by an historical event that made a connection between the physical environment and human activity. The evaluation focused on teachers’ use of assessment and student learning as a result of using the curriculum activities.  Another evaluation by GRG was a summative evaluation of Try Science, an on-line Master’s degree program in science education offered by Lesley University. The major goal of the evaluation was to examine the effectiveness of the distance-learning model by assessing outcomes for course participants that include understanding inquiry science learning and increased use of technology in the classroom.

SAE International (Formerly the Society of Automotive Engineers), Warrendale, PA

For SAE, GRG conducted a five-year longitudinal outcome evaluation of A World in Motion (AWIM), an interdisciplinary engineering supplemental curriculum kit for use in middle and high school grades that was designed to promote science, engineering, and mathematics literacy among students by engaging them in challenging engineering and design activities. Primary outcomes assessed were changes in students’ and teachers’ attitudes, interest, and knowledge.

  
EARLY CARE AND EDUCATION

Mills Consulting Group, Inc., Concord, MA

For over 20 years, GRG has carried out many early care and education studies in collaboration with our colleagues at Mills Consulting Group, Inc. (MCG), a global child care consulting firm. These have included three types of studies, as described below. For all three types, GRG participates in client meetings; consults on survey development; programs and administers online surveys, analyzes data, and consults on report preparation, including formulating recommendations

Early care and education workforce studies help states improve how they prepare, support, and reward their workforces. These typically involve surveys of child care centers as well as surveys of family child care providers. Areas of exploration include: provider experience and career information, education and professional development, benefits and compensation, and turnover intentions. Examples of clients include Early Care and Education Consortium, Washington, DC, Fairfax County (VA) Office for Children, New York State Office of Children and Family Services, Colorado Rural Resort Region, Massachusetts Department of Early Education and Care.

Child care market rate studies enable families with subsidies to enter the child-care market in a competitive position. These generally involve a survey of a sample of providers. Results are provided for the state overall as well as for local child care market areas, by type of care and age group. GRG designs and carries out the sampling plan (e.g., stratified random sampling) and enters and analyzes data (using advanced analytic techniques such as Hierarchical Linear Modeling). States for whom GRG and Mills have conducted studies include Massachusetts, Maine, Michigan, Utah, and Vermont

Child care needs assessments conducted for universities and corporations actively considering on-site child care programs, to help them respond to the child care priorities of their faculty, staff, students, and employees. Areas of exploration include: respondent demographics, interest in potential child care initiatives, and present child care arrangement. Examples of clients include: Woods Hole Oceanographic Institute, University of Washington – Bothell, Vanderbilt University Medical Center, Mountwest Community and Technical College, Concordia University, University of Pennsylvania Health System, Oregon State University, Ivy Tech Community College, and  Novartis Institutes for BioMedical Research.

Vermont Community Foundation, Montpelier, VT 

In 2009, GRG completed a study of Early Care and Education in Vermont that had the goal of exploring Vermont’s distinct needs and gathering best practices in other parts of the U.S. An informal affiliation of charitable foundations in Vermont, led by the A.D. Henderson Foundation and the Permanent Fund for the Well-being of Vermont’s Children, commissioned GRG to conduct this first-of-its-kind study of early care and education in Vermont. Through stakeholder interviews, literature review, and secondary data analysis, the study provided comprehensive, up-to-date information on the most important aspects of Vermont’s early care and education landscape, including supply and demand, early childhood spending, universal and targeted approaches to healthy child development, quality in early care and education, and education and professional development of providers. 

Families First, Cambridge, MA

Using a pre- and post-test methodology with treatment groups and waiting control groups, GRG carried out an outcome evaluation of Love and Limits, a multi-site education series that was offered to parents in the greater Boston area. The evaluation assessed the program’s impact on both the attitudes and behaviors of participating parents. The findings indicated that parents who took part in Love and Limits showed positive changes in attitude and behavior, while acquiring a better understanding about their children and themselves.

Westside Children’s Center, Los Angeles, CA

GRG conducted consultation and evaluation activities for several evaluation projects run by the Westside Children’s Center (WCC).  Activities included analysis of Desired Results for Children and Families (DRDP) parent surveys about satisfaction with the child care and development program their children attended, and analysis of surveys completed by prospective adoptive and foster families after their participation in each of a series of courses hosted by WCC staff.

WFD (formerly Work-Family Directions), Boston, MA

Over several years in the 1990s and early 2000s, GRG conducted several evaluations funded by the American Business Collaboration (ABC), a collaboration of leading U.S. companies that partnered to ensure that their employees have access to dependable dependent care programs. A broad goal of the evaluations was to use employee feedback to improve the services provided by these companies. GRG also conducted a qualitative evaluation of back-up care models in seven sites around the country.

  
ENERGY

Connecticut Energy, Hartford, CT

EEsmarts is a free science curriculum for elementary educators and middle school (through 8th grade) science educators in Connecticut based on expanding the use of energy conservation, energy efficiency, and renewable energy practices in Connecticut homes.  The program’s focus is on professional development for teachers who then introduce energy science topics to their students.  GRG’s role was to conduct a process evaluation of the effectiveness of changes the program made in response to recommendations from a 2005 program evaluation.   GRG helped document how decisions about changes were made and how these changes affected school districts and individual educators who participated in the program.

Massachusetts Energy Consumers Alliance, Boston, MA

The Renew Boston Residential Energy Efficiency Program, launched in 2010 as a partnership among the City of Boston, NSTAR, National Grid, Mass Energy Consumer’s Alliance, and Next Step Living, was designed to demonstrate how to achieve city-wide energy reduction goals. With a focus on Boston renters and homeowners living in 1-4 unit buildings, Renew Boston conducted home energy assessments in approximately 6,500 median income households. GRG’s evaluation activities were aimed at gauging the impact of the approaches used to overcome specific barriers in implementing energy efficiency upgrades identified through the home energy assessments. GRG conducted surveys and interviews with a stratified random sample of households that received a home energy assessment through Renew Boston. GRG also conducted statistical analyses of the extensive data already collected by Renew Boston in order to identify factors that explain a household’s full or partial participation in this program.

 
FELLOWSHIPS AND SCHOLARSHIPS

Brown University, Watson Institute for International Studies

GRG served for six years as the external evaluator of IGERT, An Integrated Program on Development and Inequality in the Global South, an interdisciplinary graduate fellowship program, the goal of which was to have IGERT trainees incorporate interdisciplinary and innovative techniques in their doctoral work and beyond. The evaluation consisted of formative, process, and summative components and focused on the implementation of the project as well as the success of the project in meeting its goal. Evaluation activities  included an annual survey with students and alumni and participating faculty, site visits to the program, social network analysis, and interviews with program staff. 

Council Bluffs Community Education Foundation, Council Bluffs, IA

GRG conducted a full evaluation of the Foundation’s STARS Scholarship, which aims to help residents of Pottawattamie County who have children and who work, to earn their associate’s and/or bachelor’s degrees. GRG included mixed methods in its process and outcomes evaluation, such as a review of existing program data and scholarship program processes; interviews with current staff members; online surveys of current scholars, alumni, leavers, and non-respondents; and phone interviews with a subset of the scholar sample.

Ford Family Foundation, Eugene, OR

In 2011, The Ford Family Foundation contracted GRG to evaluate its Ford Family Scholars Program, which offers college scholarships to graduating high school seniors and community college students in Oregon and the northern-most county of Siskiyou in California. In addition to the scholarship funds, the Ford Scholars Program provides support and encouragement to Scholars through conferences, counseling, and on-campus resources. GRG assessed the impact of the program through an online survey and focus groups with current scholars and their parents, with the goal of capturing the benefits of the program, exploring potential improvements, and contributing to the literature on college access. GRG also conducted a return on investment (ROI) analysis.

That evaluation has been followed by six subsequent evaluations for the Foundation, most recently of its Ford Family Graduate Scholarship Program, in which GRG conducted a survey of graduate scholars and alumni and interviews with a subset of the sample.

Institute of International Education, New York, NY

GRG conducted an evaluation of the Fulbright Foreign Language Teaching Assistant (FLTA) program for the Institute of International Education. The FLTA program brings aspiring teachers of English from around the world to the U.S. to teach their native language to American college students and to develop their own professional skills. FLTAs are embedded in 2- and 4-year colleges and universities throughout the U.S. and teach foreign language classes, lead cultural events, and enroll in classes on education theory and American history and culture. Through both quantitative and qualitative data from FLTA alumni, the supervisory staff at American colleges and universities, and the American students taught by FLTAs, we assessed participants’ levels of satisfaction with the FLTA program. We also examined personal and professional outcomes for the FLTAs, such as the extent to which they increased their English language and intercultural skills.

Gann Academy, Waltham, MA 

GRG conducted a two-year evaluation of the  Machar (“Tomorrow”) Early Career Leadership Fellowship Initiative, with funding from the Jim Joseph Foundation. This qualitative evaluation explored outcomes for the six Cohort 1 fellows who taught in pluralistic Jewish community high schools as well as outcomes for the three partnering pilot schools.

Harvard Graduate School of Education (HGSE), Cambridge, MA

GRG was contracted by HGSE to conduct an end-of-project evaluation of the HGSE Noyce Scholarship, funded by the National Science Foundation. The Noyce Scholarship encourages science, technology, engineering, and math (STEM) majors and professionals to pursue a career in K-12 math or science teaching. The purpose of this evaluation was to determine benefits and challenges of participating in the scholarship program and to assess scholar outcomes. GRG’s evaluation included secondary data analysis of existing data, a survey of all former and current scholars, and interviews with program administrators and participants.

Jack Kent Cooke Foundation, Lansdowne, VA

The Jack Kent Cooke Undergraduate Transfer Scholarship program annually awards grants designed to help high-achieving students transfer from community colleges to selective four-year schools.  GRG conducted a one-year study of outcomes for students who received a grant from the JKC or were highly competitive nominees for the grant (the comparison group), by tracing their paths and achievements in order to identify the factors that facilitate or hinder their academic and professional success.

Triangle Coalition for Science and Technology Education, Arlington, VA

The Albert Einstein Distinguished Educator Fellowship Program, then administered by the Triangle Coalition and the Department of Energy–Office of Science, offers outstanding K–12 mathematics, technology, and science teachers the opportunity to serve in the public policy arena. Fellows bring their expertise to Congress and appropriate branches of federal government and gain insights into national educational issues that can then be brought back to the classroom. GRG’s evaluation examined the impact of the program not only on the Einstein Fellows themselves, but on the federal branches in which the Fellows serve and the schools to which the Fellows return. Outcomes include the Fellows’ familiarity with national STEM policy, legislation, and funding opportunities; the value added to the host branch of having a Fellow; and the benefits to the school and district to which the Fellow returns.

WGBH, Boston, MA  

WGBH and The Partnership, Inc. developed and carried out the Next Generation Leadership initiative, which was envisioned in response to the Corporation for Public Broadcasting’s call for programming that expands the diversity of senior leadership among public broadcasters. Eight Senior Editorial Fellows were provided with a variety of skills training, mentoring, and network experience over the course of a year, with the goal of enhancing leadership skills, developing professional networks, and building pathways to executive leadership jobs. GRG evaluated the project using a pre/post/follow-up evaluation design. We used both quantitative and qualitative analysis, including social network analysis, on initial and enhanced social networks resulting from the Fellowship year.

HEALTH AND HUMAN DEVELOPMENT

Schwartz Center for Compassionate Caregiving, Boston, MA

GRG has conducted five evaluation studies for the Schwartz Center on their Schwartz Center Rounds, a multidisciplinary forum at which health care practitioners discuss difficult emotional and social issues that arise in caring for patients, with the goal of improving caregiver-patient relationships. GRG conducted a retrospective web-based survey of 256 caregivers at six experienced Rounds sites (i.e., where Rounds had been operating for three or more years), supplemented by 44 interviews with caregivers, Rounds clinical leaders, Rounds facilitators, and hospital administrators and pre-post web-based surveys of 222 caregivers from ten hospitals newly implementing Rounds.  Subsequent studies focused on different uses of the Rounds.

University of California, Los Angeles (UCLA), School of Medicine, Los Angeles, CA

A Systems Response to Improving Education on Aging in California(SAGE), a gerontology core curriculum, was developed by UCLA for gerontology, social work, and nursing programs in both two- and four-year colleges. This U.S. Department of Education FIPSE-funded project aimed to address the need to prepare a skilled and representative workforce to provide quality services and care to older Californians. Many aspects of the project were evaluated by GRG, including the management structure developed for the project, the Manual of Procedures, and the faculty development program.

As a follow up to the SAGE project, we evaluated another FIPSE-funded project, the Evidence-based Health Promotion (EBHP) Educator Certificate Program. Our evaluation questions were focused on the job market preparation and outcomes for community college students who completed the Career Technical Education (CTE) and the professional development outcomes for faculty who completed the faculty development program and taught the new certificate courses.

Dudley Street Neighborhood Initiative, Boston, MA

GRG conducted baseline and two-year follow-up surveys of the Roxbury and North Dorchester neighborhoods for the Dudley Street Neighborhood Initiative (DSNI). The work was part of the organization’s U.S. Department of Education grant to coordinate the Boston Promise Initiative, a resident-led effort designed to improve the educational and developmental outcomes of children and youth and create a community of opportunity. GRG collected and analyzed data according to several Government Performance and Results Act (GPRA) indicators. GRG  designed a survey that was fielded in the Dudley Street neighborhood by resident volunteers, who GRG trained and involved in data analysis.

AARP, Driver Safety Manual Review, Washington, DC

GRG conducted a formative evaluation of AARP’s Driver Safety Program, an in-class driving improvement course for motorists age 50 and older. The evaluation guided the AARP’s revision and development of the course and training materials. GRG was provided with a Materials Review Form from AARP for each of the three resources in their ‘field test’ version. GRG fielded the evaluation with motorists, who were asked to review each resource and complete the appropriate form, which GRG designed to obtain both objective descriptions of the information presented, and subjective comments about the instructional quality of the content.

Health Leads, Boston, MA

GRG consulted with Health Leads, an organization that mobilizes undergraduate volunteers, in partnership with providers in urban clinics, to connect low-income patients with basic resources, such as food, housing, and heating assistance. GRG identified and readied system requirements for evaluation of Health Leads’ value proposition.

Blue Cross Blue Shield of MA, Boston, MA

Trustee Advantage was a grant program designed and sponsored by Blue Cross Blue Shield of Massachusetts to help the Boards of Trustees at five Massachusetts hospitals advance the governance of quality and safety improvement at their institutions. This one-year program provided funding for three components: a coaching engagement, an in-depth experiential practicum, and participation in an ongoing peer learning community with other grant recipients. As part of our process evaluation, GRG collected data from Board members, coaches, and learning community facilitators to understand how the program was unfolding. GRG’s outcome evaluation focused on intermediate outcomes such as Board members’ commitment to improved quality and safety and understanding of their role in quality and safety governance, the extent to which Boards implemented or planned to implement changes in service of these goals, and the facilitators of and barriers to such efforts going forward.

Brazelton Touchpoints Center, Boston, MA

With funding by Robin Hood Foundation (New York, NY), GRG conducted an outcome evaluation of the Brazelton Touchpoints training program as implemented at the Harlem Hospital, New York, Children’s Zone. The Brazelton Touchpoints at Harlem Hospital program offered training to healthcare providers with the goal that practitioners integrate knowledge of child development into their practice. The evaluation included a process study of how the Touchpoints program was being implemented at Harlem Hospital and an outcome study that measured change in the behavior, practice, and knowledge of Touchpoints-trained practitioners, primarily pediatricians.

American Heart Association, Dallas, TX

GRG has conducted three evaluations for the AHA. One of these was of the educational and market needs of AHA’s Advanced Cardiac Life Support (ACLS) Program. The ACLS course was developed to provide healthcare professionals with the knowledge and skills needed for a successful resuscitation effort. The evaluation obtained feedback about the course and accompanying materials from course users as well as from AHA ACLS stakeholders. Recommendations made to AHA informed course modifications in preparation for a planned revision of the ACLS program.

LITERACY

WETA-TV, Arlington, VA

GRG conducted 10 annual evaluations of Reading Rockets, a national multimedia project that provides a variety of resources, strategies, and research-based information to help young children learn to read. Reading Rockets offers four websites, professional development for teachers, several TV programs in conjunction with PBS, and a variety of social media outlets to help build childhood literacy. In addition to annual benchmark questions, the final evaluation focused on four specific Reading Rockets resources: new Classroom Strategies videos, Literacy in the Sciences tip sheets, Family Literacy Bags, and the Start with a Book microsite. A group of active Reading Rockets users — teachers, parents, and other education professionals — provided feedback on each of these resources.

Reading Is Fundamental, Washington, DC

For this national literacy organization, GRG carried out a five-year evaluation of the RIFNet Initiative, a Star Schools distance learning project funded by the U.S. Department of Education. The broad goal of the initiative was to increase access to literacy training and content through satellites, videos, and on-line, web-based activities. GRG then completed a three-year extension of the original RIFNet evaluation, which assessed the Foundation for School Readiness program as it expanded the audience to train Spanish-speaking childcare providers and to teach family childcare providers how to engage English and Spanish-speaking families and communities in the literacy and language development of their children.

Everybody Wins USA, Washington, D.C.

GRG conducted evaluation of the Washington DC-based Everybody Wins! USA organization along with its 13 affiliates across the US, focusing in particular on the Power Lunch program. The EW! Power Lunch program brought adult mentor-volunteers (who were often supported by their places of employment to volunteer) into school settings to read books once a week with students in grades K-6. The overall goal of EW! (including Power Lunch) was to increase literacy among children who are reading below grade level and encourage love of reading. GRG’s evaluation gathered survey and interview data from several sources: national EW! professional leadership and board members, affiliate executive directors, and Power Lunch school coordinators. The evaluation helped clarify the desired outcomes, and how affiliate organizations contributed to reaching their goals.

Child Care Careers Institute, Boston, MA

GRG carried out a Boston Foundation-funded study of the literacy training needs of center-based and family child care workers in Massachusetts. Data collection instruments included both surveys and interviews, which were conducted in English, Spanish, and Portuguese. The goals of the study included assessing the need for trainings/courses to improve the job-related literacy and communication skills of family child care providers and center-based teaching staff, determining the child care workforce’s interest in receiving such training, and investigating the existence of literacy training programs already available to childcare providers.

MEDIA (INCLUDING CHILDREN’S TELEVISION)

WNET, New York, NY

In 2021-22 GRG served as research partner to the American Graduate:  Path to the Future (PTTF) Initiative during its one-year planning phase. With funding from Corporation for Public Broadcasting PTTF was designed as a “multi-platform public media initiative that will help youth in diverse communities across America explore and prepare for careers in the post-COVID-19 economy.” Emphasis was placed on “high demand careers” that pay well and that do not necessarily require a four-year college degree.

GRG conducted a baseline survey and interviews with a sample of teens across the country about their social media use and how they preferred to look for future jobs and careers. As the WNET teams produced prototypes for a web platform and videos for teens about jobs, GRG conducted iterative rounds of formative testing with groups of teens, using Zoom interviews and focus groups.

Additionally, GRG attended and presented GRG’s findings at the three sessions of the National Advisory Group and collected feedback from the members of the National Youth Advisory Board. GRG presented a final report with “lessons learned” and recommendations for the next phase of the initiative.

WGBH-TV, Boston, MA

Between 1992 and 2017, GRG conducted more than 50 evaluations of public TV programs, web sites, educational print materials, and outreach initiatives developed by WGBH.

• • Examples of cultural programs are: Oliver Twist, Anna Deavere Smith’s Fires in the Mirror, Vietnam Collection (part of WGBH’s Open Vault project), and Africans in America.
  • Examples of science programs are: NOVA ScienceNOW (five seasons), Lives in Science, Building Big, Einstein’s Big Idea, and Saved by the Sun.
  • Examples of children’s programs are: Fetch, PEEP and the Big Wide World, Arthur, Postcards from Buster, Design Squad, Arthur, Between the Lions, and ZOOM (7 seasons)

Discovery Channel, Silver Spring, MD

GRG conducted a summative evaluation of Frozen Planet, a seven-episode prime time natural history series co-produced by the Discovery Channel and BBC. The Frozen Planet project, including series and website, aimed to introduce viewers to the earth’s Polar regions and to show the impact of the year-long seasonal changes on the landscape and wildlife. GRG’s summative evaluation assessed the overall influence of the series on viewers. In particular, our evaluation examined the extent to which the series visuals and the portrayal of the natural world led viewers to understand, appreciate, and want to learn more about the environment and the global importance of the poles to the functioning of the entire Earth climate system. GRG conducted a national viewer study along with a survey of visitors to the Discovery Channel/Frozen Planet website.

Curved Light Productions, East Hampton, NY 

GRG conducted summative evaluation of StarTalk Radio, an innovative new genre of science radio hosted by astrophysicist Dr. Neil deGrasse Tyson that bridges the intersection between popular culture and science education. The radio programs combine comedy, references to pop culture, and public fascination with space science to reach an untapped audience for the informal science field.  Intended outcomes of the project included increasing knowledge of and interest in space science topics, and motivating audiences to pursue additional learning activities as the result of listening to the programs. GRG’s evaluation examined how well the joint presentation of science and humor reached a public audience and how well it engaged and informed them and piqued their interest such that they are motivated to continue learning more.

The HistoryMakers, Chicago, IL

GRG conducted process and outcome evaluation of The HistoryMakers Digital Archive project, funded by the PwC Foundation. The HistoryMakers transcribed and processed 2,000 interviews that were not yet accessible to the public via The HistoryMakers Digital Archive. The goal was to increase access for students and teachers in public and charter schools to the resources and lesson plans afforded through these primary source materials. GRG conducted a case study with three phases of external evaluation intended to assess the effectiveness of the fully populated Digital Archive (containing full transcripts of all interviews). The three phases included: key stakeholder interviews; process evaluation, including site visits, and classroom observations, and outcome evaluation, including data from three years of teacher and student surveys.

Kikim Media, Menlo Park, CA

Science Bytes, produced by Kikim Media, was a six-part video series that brings the latest scientific research to the general public through online videos. GRG worked with Kikim Media on this formative evaluation to assess the appeal, interest and increased understanding of these short science videos.

KCET-TV, Los Angeles, CA

GRG conducted a national summative evaluation for KCET’s new children’s science program, Sid the Science Kid. The daily series and associated national outreach aim to engage children ages 3-6 years old and their caregivers. For children, goals include supporting and extending their natural science-related curiosity by increasing science process skills they can use to seek answers as they explore the world around them. For caregivers, goals include increased interest, comfort, and confidence to support their children’s curiosity. GRG’s evaluation examined the effectiveness of the series and website through a viewer study to compare science-related interests, attitudes, and activities among those who were or not exposed the series and website. We also assessed the implementation and effectiveness of the community outreach, including reach, use, and satisfaction with the outreach trainings and materials.

Screenscope, Washington, DC

GRG carried out several evaluations for the documentary makers at Screenscope. One was an evaluation of an educational outreach initiative for Season Two of the public TV series Journey to Planet Earth. The series explores the fragile relationship between people and the world they inhabit. The series was accompanied by an extensive outreach initiative with 11 museums around the country that conducted community events involving teachers, students, and the general public. The evaluation explored the overall effectiveness of the new series and outreach, including collaborative partnerships, the informal science activities, and influence of the series on viewers.

Digital Diaspora, New York, NY

Family Pictures USA (FPUSA) is a television series and transmedia project that explores neighborhoods and cities through the lens of the family photo album. FPUSA was created, co-produced, and hosted by Thomas Allen Harris, with funding from the Corporation for Public Broadcasting. A primary goal of the project was to enlarge participants’ and viewers’ understanding of history, diversity, and shared values by exploring neighborhoods and cities through the lens of the family photo album.  For the participating local PBS stations, a goal was to encourage and empower them to engage with their audience and their partners, and to collaborate in innovative ways with each other and with independent filmmakers. GRG evaluated the project through the following:  observation of photo-sharing events in southwest Florida and Raleigh, NC; conducting exit surveys and then follow-up surveys with the event attendees; fielding an online survey with series viewers; and three collaboration inventory surveys with the project team and partnering collaborators. 

 
MUSEUM AND LIBRARY PROGRAMS

American Museum of Natural History (AMNH), New York, NY

GRG conducted evaluation of three educational components resulting from AMNH’s collaboration with NASA: (1) the 40-minute space show, The Search for Life: Are We Alone?, (2) three Science Bulletins housed within three permanent exhibits (these include large-screen projections of high-definition video and 3-D computer), and (3) three middle school curriculum units (Biodiversity, Antarctica, and Deep Sea Vents). The evaluation sought to determine the extent to which each educational component succeeded in meeting the goals set forth by the museum.

Mashantucket Pequot Museum and Research Center, Mashantucket, CT

GRG provided evaluation and consulting services for the continuation and completion of an evaluation of the museum’s Institute of Museum and Library Services-funded National Leadership Grant. These grants aim to help libraries and museums develop models and produce results that help to foster individual achievement, community responsibility, and lifelong learning. GRG’s role was to conduct a series of interviews with the key players in the project in order to learn about the outcomes of the project based on the perspectives of museum staff, partner site staff, and partner site interns.

Dartmouth College, Hanover NH

Rural Gateways is designed to enhance rural librarians’ self-efficacy and professional self-identification as informal science facilitators through professional development, an online community of practice, and scaffolded experiences of providing adult STEM programming in their communities, with the eventual goal of building their capacity to sustain this programming beyond the life of the project. Dartmouth College is collaborating on this NSF-funded grant with Dawson Media Group, Oregon State University, the Association of Rural and Small Libraries, and the Califa Library Group. GRG’s summative evaluation focuses on the quality of the professional development model, the success of the adult science programming, and changes over time in rural librarians’ self-efficacy and identity as informal science facilitators. Our evaluation is one component of a larger research project funded by NSF AISL that will inform the field of informal science learning, with particular emphasis on how to reach new and underserved audiences.

Previously, GRG evaluated the team’s efforts on Pushing the Limits, an NSF-funded partnership to build rural and small libraries’ capacity to enhance public understanding of science and math. The team created discussion guides and specially produced video segments for four science café-style guided public discussions for an adult audience. The project centered around the notion that we all use science in our everyday lives to “push limits.” Each of the four units was also linked to a popular work of literature. Rural librarians received professional development to facilitate the public events in partnership with a local science advocate. Twenty rural and small libraries were selected to pilot the Pushing the Limits program, after which it was expanded to 100 additional rural and small libraries throughout the U.S.

Harvard Smithsonian Center for Astrophysics, Science Media Group, Cambridge, MA

GRG conducted a summative evaluation of The Black Hole Experiment Gallery, a traveling exhibit and accompanying materials produced by the Smithsonian Astrophysical Observatory of the Harvard-Smithsonian Center for Astrophysics. Two innovations of this project that were explored through evaluation activities were: (1) the inclusion of significant input from youth collaborators in the design and development phase, and (2) the use of networked exhibit technologies to personalize and enhance the visitor experience of science inquiry both within and beyond the exhibit gallery. The adult and youth collaboration was evaluated through student focus groups and adult interviews. The summative exhibit evaluation included exit interviews, surveys, observations, and analyses of data gathered through the networked technology.

United States Holocaust Memorial Museum, Washington, DC

GRG conducted an evaluation of the Arthur and Rochelle Belfer Conferences. The three-day summer conferences offer participating middle school and high school educators the opportunity to learn the history of the Holocaust as well as methods and classroom strategies from leading scholars, master teachers, and museum educators. We examined the influence of the conference on participants’ formal classroom (or informal) instruction, and knowledge about the Holocaust, through surveys of past program participants, observing the conferences, and surveying participants before and after conference.

USS Constitution Museum, Charlestown, MA

The USS Constitution Museum’s new Forest to Frigate exhibition focuses on the people and processes behind the building of one of Boston’s most iconic landmarks, the USS Constitution. GRG conducted a visitor study to measure the impact of viewing the exhibition on visitors’ understanding of the construction of this ship. Using intercept surveys and interviews with museum patrons, we also explored the extent to which Forest to Frigate promotes engagement and personal connections with the story of the USS Constitution.

WGBH-TV, Boston, MA

WGBH Boston created a national outreach initiative for We Shall Remain, a five-part documentary series on Native American history produced for American Experience. Fifteen public TV stations convened coalitions of local community organizations to create materials and events that made local connections to the national series. Then, in partnership with the American Library Association, WGBH developed an event kit with ideas and resources to help libraries organize events related to the series. GRG’s evaluation described how the We Shall Remain community coalition and library outreach activities unfolded and assessed their reach and impact. Outcomes included the perceived benefits of coalition outreach activities to coalition members, the organizations they represent, and the local community. For the library event kits, outcomes included librarian perceptions of the impact of the materials and activities on library patrons.

Califa Library Group, San Mateo, CA

Public Libraries Advancing Community Engagement (PLACE):  Environmental Literacy through Climate Change Discussions  provided librarians with professional development opportunities that enabled them to further their presentation and facilitation skills, as well as gain greater familiarity with climate science content. They then co-facilitated three public library programs in their communities, in which participants read popular fiction and non-fiction books and watched human interest videos about real people who have developed resilient strategies for facing climate challenges. Through exploration and discussion, participants were introduced to online tools that can provide information for their own community, family and individual planning for extreme weather and other climate events. GRG’s evaluation assessed librarians’ gains in their own knowledge of  –and their ability to support patrons in using — climate resilience tools, awareness of local resilience efforts, and likelihood of involving their libraries in those efforts. The evaluation also assessed patrons’ knowledge gains.

The Franklin Institute Science Museum, Philadelphia, PA

The Franklin Institute partnered with the Center for Neuroscience & Society at the University of Pennsylvania to create Neuroscience in Your World, an NIH-funded suite of K-12 programs related to the museum’s 8,500-square-foot Your Brain exhibit to engage students and teachers in learning about neuroscience and its relevance to daily life. GRG provided formative and summative evaluation of a day-long professional development session for K-8 teachers, an all-day experiential field trip package for grades 5-8, a neuroscience mini-course/curriculum units for grades 9-10, and a semester-long course on neuroscience and society for grades 11-12.

  
PARTNERSHIPS AND COLLABORATIONS

Boston Children’s Museum, Boston, MA

GRG is playing a consultative role in the evaluation of the Institute of Museums and Library Services-funded project, School Readiness through Partnerships:  Museums, Libraries, and State Early Learning Systems Project.  A major goal of the project is to develop and share early education resources (trainings, kits, and guides) with partnering informal education organizations and to create and implement a Hub and Partnership network infrastructure. These resources and network together will work to expand and enhance partnerships among community organizations in Massachusetts. GRG is participating in high-level discussions about the project’s desired impacts, as well as providing consultation to the project’s internal evaluator on evaluation design, instrument development for the three levels, data collection strategies, data analysis (including social network analysis), and reporting of results, conclusions, and recommendations. One of the goals of GRG’s consultation is to help the project staff further develop institutional capacity in program evaluation.

Our consultation is drawing upon GRG’s previous experience evaluating BCM’s Museums and Libraries Project.  For that Race to the Top grant, GRG conducted statistical analyses on the existing surveys for which BCM had data from museums and libraries, conducted a simple analysis of a past evaluation form, and developed and fielded an online survey to recipients of each of the four kits developed by BCM in order to determine their effectiveness.

Education Development Center, Inc. (EDC), Waltham, MA

GRG has been evaluating EDC’s STEM Learning and Research (STELAR) Center since it was first funded by the National Science Foundation (NSF) in 2013. STELAR is the Resource Center serving NSF’s Innovative Technology Experiences for Students and Teachers (ITEST) community. ITEST supports the design and research of innovations that ignite interest in and prepare PreK-12 youth for future STEM and ICT careers. STELAR provides technical assistance to potential and funded projects, disseminates findings, and helps broaden ITEST participation to investigators, audiences, areas, and institutions not currently represented in its portfolio. GRG’s evaluation is designed to answer questions and promote strategic learning about the progress and impact of STELAR’s activities. The evaluation uses a value creation framework to help STELAR make decisions about how best to create value for stakeholders.

MetroWest Health Foundation, Framingham, MA

GRG conducted process evaluation of a nine-town tobacco education collaboration in west suburban Boston, funded by the MetroWest Health Foundation. Our evaluation examined the extent to which the coalition was able to accomplish its objectives of having local public health departments working together on this public health issue; the challenges and barriers to this work; level of effort across the coalition; and best practices that can be applied to similar work on other public health issues. Data collection methods include document review and semi-structured interviews with key stakeholders and program staff.

United Way of Massachusetts Bay and Office of Boston Mayor Menino, Boston, MA

GRG completed a year-long process evaluation to document Boston’s city-wide Birth to Five School Readiness Initiative. This initiative aimed “to develop a comprehensive vision of child, family, and community school readiness.” GRG’s work documented how individuals from different stakeholder perspectives came together to create a collaborative, sustainable model for school readiness. We participated in all aspects of the planning initiative, including working closely with the Mayor’s Advisor/Project Manager and other initiative leaders. GRG collected both quantitative and qualitative data from the initiative planning meetings, web-based surveys of all initiative members, key informant phone interviews, and a review of other select cities’ early childhood initiatives. The evaluation provided ongoing feedback during the year-long process, and then summatively answered key questions about the process and outcomes.

  
PROFESSIONAL DEVELOPMENT

EDC, Waltham, MA

Funded by the National Science Foundation, EDC led the Massachusetts Exploring Computer Science Partnership (MECSP), which aimed to expand computer science education in high schools across Massachusetts to engage greater numbers and greater diversity of students in computer science. Educators who taught the Exploring Computer Science (ECS) course received professional development delivered through a train-the-trainer model, which has created a cadre of Teacher Leaders prepared to teach ECS and facilitate future professional development institutes. GRG’s evaluation of MECSP assessed the implementation, effectiveness, and impact of the professional development delivery for new ECS teachers and course impact on enrolled students’ self-efficacy, interest in CS, and understanding of connections between CS education and career opportunities.

Jim Joseph Foundation, San Francisco, CA & Combined Jewish Philanthropies, Boston, MA  

GRG served as external evaluator of B’Yadenu (Hebrew for “In Our Hands”) Demonstration Project, Meeting the Needs of Diverse Learners in Jewish Day Schools: A Whole School Approach, which was funded by the Jim Joseph Foundation and spearheaded by CJP. The B’Yadenu model is designed to establish a comprehensive, sustainable professional development model for Jewish day school educators, building the long-term capacity of teachers and leaders to better serve students with a range of learning needs. The project was originally carried out in five day schools in the greater Boston area.

GRG’s initial four-year evaluation assessed the model’s effectiveness at implementing “whole school change” and provided recommendations on implementing the program in other communities. Evaluation methods included online surveys, site visits, focus groups, timed classroom observations, observations of professional development sessions, and discussions with each school’s leadership team. The evaluation results led the foundation to continue funding the initiative for another three years. At the end of that follow-on period, GRG conducted a qualitative evaluation of continued progress made by the five schools; evaluation methods were focus groups with teachers and teacher-leaders in each of the five demonstration schools, participating in final leadership team group interview at each school, and participating in the wrap-up convening of the schools and project team. In addition, GRG interviewed teacher-leaders in WAshington, D.C.,Los Angeles, and Washington, D.C., where a small number of schools sought to adapt the B’Yadenu model.

Paul G. Allen Foundation, Seattle, WA

Success at the Core (SaC) is a web-based professional development toolkit – with videos as the centerpiece – which builds on research that shows how instruction, school leadership, and learning culture and community have an impact student outcomes. Launched as a partnership between EDC and Vulcan Productions, with additional funding from the Stuart Foundation and the Paul Allen Foundation, SaC was designed to allow schools and teachers to create their own professional development experience customized to meet their own needs. GRG’s evaluation activities were aimed at gauging the impact of toolkit use on the quality of instruction and, subsequently, on student achievement. To this end, GRG conducted teacher and student surveys and observations and statistical analyses of existing student achievement data.

EDC, Waltham, MA

GRG conducted a process and summative evaluation of Taking NPASS to Scale (NPASS2): Creating State-based Professional Development Networks for Out-of-School-Time Science, a program developed by the Center for Science Education (CSE) at EDC. This three-year NSF-funded program was designed to provide professional development training to science trainers to increase Out-of-School Time (OST) science programming for children. The program focused on underserved populations in eight geographically and demographically diverse U.S. states. The summative evaluation examined how the project built the states’ infrastructure and capacity to support this work into the future.

PUBLIC ENGAGEMENT WITH SCIENCE

Space Telescope Science Institute, Baltimore, MD

GRG clients from the NASA’s Universe of Learning (UoL) project were among those selected to implement the next phase of NASA’s Science Activation (SciAct) program. NASA’s UoL is a partnership between the Space Telescope Science Institute, Caltech / IPAC, Center for Astrophysics | Harvard & Smithsonian, and NASA Jet Propulsion Laboratory. In 2016, NASA’s UoL was one of 27 cooperative agreements selected by NASA’s Science Mission Directorate to implement a new strategic approach to more effectively and efficiently incorporate NASA science experts and content into the learning environment for learners of all ages. In September 2020, the NASA’s UoL project was one of 21 of the original awardees, along with nine new awardees, selected for funding through 2026. The multi-institution team works together to create and deliver a suite of astrophysics-themed products, programs, and professional learning experiences for informal and life-long learning environments.

GRG supports the team through strategic learning and evaluation. Program evaluation includes evaluation of the NASA’s UoL overall portfolio, as well as evaluation of individual NASA’s UoL projects. Different types of evaluation are conducted based on the development stage of the program and of individual products, resources, and experiences. Key types of evaluation include needs assessment, formative and process evaluation, summative evaluation, and portfolio analysis. GRG also supports the NASA’s UoL team in SciAct Program activities as appropriate, and participates in SciAct Program portfolio-level evaluation activities.

University of Massachusetts-Boston, MA

GRG is currently evaluating Cool Science, a five-year NSF-funded project integrating science and art to promote youth learning about the science of extreme weather in Massachusetts and Kansas and Missouri. Cool Science provides professional development to informal educators, who work with youth to create artworks submitted to a competition to be displayed on regional buses and in art exhibitions to promote adult learning. GRG is assessing the effectiveness of the mentor workshops, the bus displays of youth art, and the annual celebrations in promoting engagement and learning around extreme weather. Evaluation activities include mentor surveys and focus groups, in-person bus rider surveys, and observations of the art exhibitions, as well as surveys of exhibition participants.

Center for Aquatic Science at Adventure Aquarium, Camden, NJ

In the Communities of Learning for Urban Environments and Science (CLUES) project, funded by NSF’s Informal Science Education program, four museums collaborated to build community capacity for STEM education and increase STEM learning opportunities for underserved families. CLUES provided apprentices from the community with intensive professional development in informal science education; in turn, apprentices trained and mentored  presenters in providing accessible family education programs on neighborhood environmental issues. GRG’s process evaluation focused on the implementation of the apprentice and presenter professional development (PD) activities and the family STEM education programs. GRG also evaluated outcomes, including community STEM education capacity; apprentices’ and presenters’ knowledge, skill, confidence, and interest in informal science education; and families’ knowledge of and connections to their urban environments.

University of California, San Diego, San Diego, CA

GRG served as the external evaluator of the three-year NSF-funded Science Festival Alliance (SFA), a collaborative initiated  by the University of California San Diego, the MIT Museum (Cambridge, MA), the University of California San Francisco, and The Franklin Institute (Philadelphia, PA). The focus of the SFA over its first three years was helping establish and sustain science festivals in each of these four cities. GRG conducted a multi-method process and summative evaluation to assess the success of the SFA project at meeting its intended impacts. Evidence of success was gained largely through surveys of public and professional audiences at the festivals as well as SFA document review, interviews with SFA science festival team members, and participatory observations at SFA meetings. Our evaluation answered questions about who participated in science festivals, the benefits of science festivals (for attendees as well as participating STEM practitioners), the most important characteristics of science festivals (that were related to increased benefits for attendees), and the support that is needed to initiate and sustain science festivals.

University of Massachusetts, Lowell, Lowell, MA

Led by researchers at UMass Lowell’s Graduate School of Education, Science2Go was an NSF-funded public education project to provide information about climate science on Boston’s city-wide Mass Transit Authority (MBTA) subway placards and platforms, and on commuters’ smartphones, in an effort to teach the public about the issue. GRG’s summative evaluation focused on the impact of Science2Go on riders’ awareness and understanding of climate change as well as their motivation to take action.

Colorado State University, Fort Collins, CO

Housed at Colorado State University and supported by the NOAA and NSF, the Community Collaborative Rain, Hail & Snow Network (CoCoRaHS) is a non-profit, community-based network of volunteers working to measure and map precipitation across the United States. GRG helped CoCoRaHS explore opportunities to expand their educational and citizen science initiatives by conducting a library needs assessment, a formative assessment of the “CoCoSchools” curriculum, and an evaluation of the effectiveness of short animated videos aimed at teaching adults about climate and weather-related topics. Data collection activities include online surveys of library staff and teachers and a two-group randomized comparison study of the impact of animated videos versus traditional webinars using 100 adult participants recruited from the general public.

Space Telescope Science Institute, Baltimore, MD

The Office of Public Outreach (OPO) at Space Telescope Science Institute (STSci) conducts public engagement activities for NASA’s Hubble Space Telescope (HST), the upcoming James Webb Space Telescope (JWST), and the future Wide Field Infrared Survey Telescope (WFIRST). With current interest to develop metrics for the effectiveness, reach, and impact of OPO’s public engagement activities, GRG conducted initial background research and data collection at public events. In 2018, GRG assessed STSci’s booth at the USA Science and Engineering Festival. In 2018-19, GRG continues to evaluate local and national events.

 
STRATEGIC PLANNING AND EVALUATION CAPACITY BUILDING

American Evaluation Association (AEA), Washington, DC

GRG was proud to be selected from among 24 bidders to work with our own professional association to provide AEA’s board with a better understanding of its membership to facilitate strategic planning and growth of the organization. We conducted an online survey of all AEA members, including international members, as well as interviews and an online focus group. In addition to presenting to the board, the results were widely disseminated to the general membership via presentations, reports, and online resources.

Anti-Defamation League (ADL) New England Region, Boston, MA

GRG assisted ADL in strategic planning by working with the Education Department to develop a logic model of its Peer Training Program, used in schools around the country to combat bullying.

New England Science and Sailing Foundation (NESS), Stonington, CT

GRG is engaged in a collaborative, participatory evaluation consultation with NESS, a non-profit ocean adventure education program focused on the values of experiential learning, personal growth, inclusiveness, and stewardship. GRG has assisted NESS in developing an evaluation framework, building evaluation capacity among NESS staff, and sharing results with funders, partners, and other stakeholders. Specific activities included reviewing the organization’s evaluation efforts, analyzing historical data, refining logic models, supporting implementation of specific evaluation plans, and helping determine and summarize program outcomes.

  
YOUTH PROGRAMMING

Boston Harbor Now, Boston, MA

GRG conducted an evaluation of the Boston Harbor Islands (BHI) Youth Engagement project, a two- year joint initiative of Boston Harbor Now, Inc. (BHN), Save the Harbor/Save the Bay (SHSB), the Massachusetts Department of Conservation and Recreation (DCR), and the National Park Service (NPS), with funding from the Massachusetts Environmental Trust (MET). The evaluation consisted of three main components:  (1) development of a logic model for the Youth Engagement Plan and a situation analysis to better understand who is being served by existing programs and assess how the programs could be improved; (2) formative evaluation of the Boston Harbor Islands Summer 2016 Discovery Camp, which included camp observations, student surveys, and discussions with youth service providers; and (3) summative evaluation of the Summer 2017 Discovery Camp that incorporated observations, along with student, parent, and youth provider surveys.

International Rescue Committee (IRC), New York, NY

GRG evaluated the IRC’s Education and Learning (E&L) youth programming: Refugee Youth Summer Academy (RYSA), Out-of-School Time programs (3 after school and one on Saturday), Academic Coaching, and Leaders in Training (LIT). The IRC in New York’s E&L unit supports the personal growth, cultural adjustment and educational development of refugee youth, empowering them to attain high school graduation, enroll in post-secondary education, and become active participants in their new communities. The evaluation was designed to provide information to be used by the E&L staff to make strategic decisions regarding program improvements and program funding. The evaluation results also informed the development of a 3-5 year vision and plan for youth programming.

FIRST (For Inspiration and Recognition of Science and Technology) Manchester, NH

GRG worked with FIRST in several capacities:  creating internal feedback mechanisms for the FIRST Robotics Competition, in which teams of high school students, educators, and engineers from sponsoring corporations, design and build robots for various competitions; conducting external evaluation of FIRST LEGO® League, a high-tech sporting event in which 4^th through 9^th grade students design, build, and program robots competing for the winning solution, using the LEGO® MINDSTORMS™ robotics invention system; and consultation regarding the FIRST LEGO® League inner city initiative in Detroit, in which we explored increases in participation among inner city children, the effectiveness of the formal mentoring process, success of local fundraising efforts, and ability of teams to self-sustain beyond the initial project period.

Massachusetts Institute of Technology (MIT), Cambridge, MA

GRG conducted a multi-year process and outcome evaluation of Terrascope Youth Radio, an NSF-funded project in which urban teens developed, reported, wrote, produced, and hosted a regularly broadcast radio program on environmental and Earth-system science. The process evaluation component documented how the initiative unfolded and operated. The outcome component evaluated consequences for the beneficiaries of the initiative, including local youth participants, youth radio participants nationwide, and youth radio listeners. Participating teens reported on science topics that are engaging and relevant to their lives at the same time as they learned radio production skills. The initiative aimed to serve as an impetus to increase coverage of environmental science and other STEM topics and to establish a model for university researchers and students to engage and work with youth in their communities.

National Cancer Institute (NCI), National Institutes of Health, Bethesda, MD

GRG conducted a controlled, randomized, seven-year (1998-2005) longitudinal evaluation of  Science Enrichment Program (NCI SEP). As a five- to six-week residential summer program for 50 post-9th grade students from minority populations underrepresented in the biomedical sciences, the program was offered at two sites (University of Kentucky and San Diego State University). The goals of SEP were to broaden students’ sociocultural backgrounds and encourage them to select science, math, research or health careers. The evaluation collected both quantitative and qualitative data from multiple sources (i.e., students, parents, and program staff) using surveys, interviews, and annual site visits to each regional program.

Imaginary Lines Inc./Sally Ride Science, San Diego, CA

GRG served as the external evaluator for Imaginary Lines, Inc./Sally Ride Science national, annual TOYchallenge, designed for 5^th through 8^th graders who participate in engineering design teams. GRG’s outcome evaluation included pre-, post-, and follow-up surveys of TOYchallenge participants and coaches, as well as observations of teams and event