Standardized Assessment Measures at Shepherd University

Shepherd University has utilized several nationally certified standardized assessment measures to gain a better understanding of issues related to student learning and performance. Below we've described some of these tests in detail.

THE MEASURE OF ACADEMIC PROGRESS AND PROFICIENCY (MAPP)

The MAPP is “a standardized measure of college-level reading, mathematics, writing, and critical thinking in the context of the humanities, social sciences, and natural sciences.”  The exam is designed by the Education Testing Service (ETS).  According to ETS, “The MAPP test is designed for colleges and universities to assess their general education outcomes, so they may improve the quality of instruction and learning.  It focuses on the academic skills developed through general education courses, rather than on the knowledge acquired about the subjects taught in these courses.”  It is a multiple choice examination. 
        Shepherd University has used the MAPP to assess general education outcomes in the following years: 2006, 2007 and 2010.  During the years that the MAPP was not conducted Shepherd University used another general education outcome assessment: the Collegiate Learning Assessment (CLA).  A sample selection of students completed the Abbreviated Form of the MAPP each year it was conducted.  From 2006 through 2010, 235 students have completed the MAPP; 83 students in 2006, 97 students in 2007, and 55 students in 2010.  The Abbreviated From of the MAPP contains fewer questions than the Standard Form and takes less time for students to complete.  The Abbreviated Form contains 36 questions and students are given 40 minutes to complete those questions.  Although the Abbreviated form is shorter it has been psychometrically tested and demonstrated to assess, according to ETS, “the effectiveness of general education programs to meet the requirements for accreditation and performance funding.”  The MAPP can be used to inform teaching and learning at an institution through the use of the score reports that display any areas of weakness.
      
The MAPP scores on the Abbreviated Form can range from 400-500 and, as with any standardized exam, there is always a Standard Error of Measurement (SEM).  The SEM is the score plus or minus a value that is used to identify confidence in the standardized exam’s statistical properties.  Because the Abbreviate Form is used to test smaller samples the scores are only reliable for specific uses.  For example the scores on the Abbreviated Form of the MAPP can be used to evaluate curriculum, but are not appropriate for evaluating teachers or for group to group comparison.  It is also important to understand that in smaller samples the mean score can be greatly affected by one or two scores that are very high or very low.  In 2006 the mean score for Shepherd University students on the MAPP was 442.5, with a Standard Deviation (SD) of 16.47.  In 2007 the average score for Shepherd University students on the MAPP was 442.3, with a SD of 16.23.  When examining the mean scores of other Baccalaureate (liberal arts) colleges whose students have completed the MAPP it was found that 53% of all of these colleges had average scores below the average scores at Shepherd University in 2010.  The mean score on the MAPP for Shepherd University students in 2006 and 2007 was above the national average score for all Baccalaureate colleges in 2010, which was 441.17, with a Standard Deviation of 9.8. In 2009 the MAPP was completed by first year students and the average score was 439.08, with a SD of 17.64.  In 2010 the MAPP was completed by senior students and the average score was 442.89, with a SD of 17.58.   The average score for the first year students in 2009 was below the National average for baccalaureate colleges in 2010, but when comparing the National average score for first year students in 2010; 436.2, SD = 8.21, it is found that the mean score for Shepherd’s first year students in 2009 is higher.  In the National first year student data for 2010 it was found that 61% of institutions had average scores below Shepherd.  When comparing the first year students’ average scores in 2009 to the senior students’ average scores in 2010 it is found that senior students had a higher average score.  The senior students should have higher scores than first year students because this difference demonstrates that Shepherd students learn the important skills that are measured by the MAPP, such as critical thinking, mathematics, reading and writing.  The National data for baccalaureate college seniors in 2010 shows that the mean was 445.89 with a SD of 9.27, and that 32% of institutions had average scores below Shepherd. The National baccalaureate mean is slightly higher than the mean for Shepherd seniors in 2010, but the National data also has a smaller SD, which indicates that there is less variance in the national scores.
It is important to note that some things can have a great effect on the mean scores for the MAPP.  Students who score very high or very low have scores that are significantly different from the mean and their scores are called outliers.  Outliers will also affect the Standard Deviation (SD).  The SD represents how well the data (scores) are clustered around the mean.  This indicates that a larger SD represents that the data are more spread out and that there are outliers in the data set.  The SD for Shepherd students in the years 2006 and 2007 is smaller than the SD found in the 2009 and 2010 data.  All four data sets have relatively high SD’s, which indicates that some students do very well on the MAPP, while others may not do well, and still others do marginal work.
         
With each score report there is a chart included with a summary of the Proficiency Classifications.  This chart displays the percentage of students who were classified as the following; proficient, marginal, or not proficient, in six different categories of the MAPP.  The six categories assessed are as follows; critical thinking, reading level 2, reading level 1, writing level 3, writing level 2, writing level 1, mathematics level 3, mathematics level 2 and mathematics level 1.  A student is classified as ‘marginal’ if their test results do not provide enough evidence to classify the student as proficient or not proficient.
       
In comparing the percentage of proficiency levels from the 2006 to the 2007 administration of the MAPP there are some differences.  The percentage of students classified as proficient, marginal, and not proficient, in Critical Thinking for 2006 and 2007, did not change: 6% were proficient, 12% were marginal, and 82% were not proficient.  For Reading (level 2) there was a small difference in the percentage: 2006; 43% proficient, 17% marginal and 40% not proficient; 2007; 35% proficient, 27% marginal and 37% not proficient.  For all other areas assessed by the MAPP there are small percentile differences.  The percentile differences between the 2006/2007 data and the 2010 National data for the MAPP are slightly larger.  For example, Shepherd University’s 2006 percentage of students who are classified as proficient is larger than the National (Baccalaureate colleges) data in the areas of critical thinking, writing level 2, and math level 1.  For 2007 the percentage of students classified as proficient is larger than the National (Baccalaureate colleges) data for the areas of critical thinking, reading level 2, reading level 1, writing level 2, and writing level 1.  Our data show that critical thinking is an area that needs improvement at both Shepherd University and across all Baccalaureate colleges in the U.S.  It is also evident that more difficult levels of writing and mathematics could use some improvement. 

THE COLLEGIATE LEARNING ASSESSMENT (CLA)

The Collegiate Learning Assessment (CLA) is an innovative approach to assessing the institution’s contribution to student learning and is developed by CAE with the RAND Corporation. Its’ measures are designed to simulate complex, ambiguous situations that every successful college graduate may one day face. Students are asked to analyze complex material and provide written responses. The CLA measures are uniquely designed to test for reasoning and communications skills that most agree should be one outcome of a college education.
In order to assess the institution cross-sectionally, we test a sample of first year students in the fall and a sample of seniors in the spring. We receive reports that look at how our freshman class compares to other CLA participants (adjusted for SAT or ACT scores). After testing seniors in the spring, we receive a full Institutional Report that evaluates our schools value-added on a comparative basis. Testing every year allows us to measure for effects of changes in curriculum and compare the results nationally, as well as suggest ways to utilize the CLA results. A summary of Shepherd's CLA results throughout the past several years is included below. For more specific information, you may open the .pdf links below and view institutional reports or view our most recent scores at www.collegeportraits.org.

Shepherd University Collegiate Learning Assessment Scores:

The most important factor to look for in the CLA is the “value added.”  This value is assigned after adjusting for entering academic ability of students who take the exam.  The difference, then, between performance of seniors and first-year students is known as the “value added.”

2005-2006 Report

Our value added is “at expected.” 
N= 79 Freshmen
N= 72 Seniors

2006-2007 Report
Our value added is “well above expected.”
N= 78 Freshmen
N= 79 Seniors

2007-2008 Report
Our value added is “above expected.”
N = 71 Freshmen
N = 109 Seniors

2008-2009 Report
Our value added is “well above expected.”
N= 106 Freshmen
N= 108 Seniors

2009-2010 No Report due to CAE Scheduling Issues

2010-2011 Report
Our value added is “at expected.”
N= 71 Freshmen
N= 62 Seniors

2011-2012 Report
Our value added is “at expected.”
N= 70 Freshmen
N= 74 Seniors

 

THE NATIONAL SURVEY OF STUDENT ENGAGEMENT TEST (NSSE)

What is NSSE?

Since 2000, the annual NSSE “College Student Report” survey has been used to collect information from undergraduates at four-year colleges and universities across the nation. Its purpose is to assess the extent to which students engage in educational practices known to correlate with success in learning.
 
Launched by a grant from The Pew Charitable Trusts, the project is now supported by institutional participation fees and is cosponsored by The Carnegie Foundation for the Advancement of Teaching and the Pew Forum on Undergraduate Learning. NSSE is administered by the Indiana University Center for Postsecondary Research in cooperation with the Indiana University Center for Survey Research.

What sorts of questions comprise NSSE?

There are five primary areas that are assessed by the NSSE. They are referred to as Benchmarks of Effective Educational Practice and are:

  • Level of academic challenge
  • Active and collaborative learning
  • Student-faculty interaction
  • Enriching educational experiences
  • Supportive campus environment

Who participates in NSSE?

More than 393,500 first-year and senior students participated in the NSSE 2010 sample. These students were randomly selected from data files provided by 595 participating institutions in the US and Canada.


 The 2010 Shepherd sample included students from all schools across campus that have first-year and senior students. The total number of respondents was 559 or 35% of all Shepherd University freshmen and seniors. All of the Shepherd respondents (100%) completed the survey via the Web. The summary of the 2010 report focuses on statistically significant differences between Shepherd and results from three comparison groups: other Southeast Public education institutions, other institutions that hold the same Basic Carnegie Classification (Baccalaureate – diverse fields) as Shepherd, and the overall results nationwide from NSSE 2010 respondents. 

One of the greatest improvements found in this past year’s survey is that all deviations from the norm were found to be small (S) in significance based on effect size (below .4). This was not the case in 2005 where many negative deviations were moderate or even large. A summary of the comparison between the 2005 and 2010 NSSE data is presented below. More specific data links are also included below and information on our NSSE scores may be found on www.collegeportraits.org.

Specific areas of improvement in 2010 gleaned from this assessment (as compared with the 2005 data) include the following:

1.  In 2005, first-year students and seniors both reported being less likely (S) to work with a classmate outside of class to prepare for class assignments than was common for the PG, BG or national norm. In 2010, Shepherd’s reported results were comparable to all NSSE comparison groups.

2.  In 2005 both first-year and senior Shepherd students gave fewer (M) class presentations than BG students, while Shepherd seniors also gave fewer (S) presentations than reported by the PG and national comparison groups.  In 2010 only senior students reported giving fewer (S) class presentations than was common for CC peers.
                       
3.  In 2010, senior students at Shepherd reported a greater (S) opportunity and likelihood to participate in a community-based project (e.g. service learning) as part of a regular course, than was reported by senior respondents at other SP and nationwide institutions.

4.  In 2005, Shepherd first-year students reported a greater likelihood (S) to be asked to memorize facts, ideas or methods and be able to repeat them in the same form, than was reported by BG or overall first-year students. In 2010, Shepherd’s reported results were comparable to all NSSE comparison groups.

5.  In 2005, Shepherd first-year students were less likely (S) to analyze ideas in depth or synthesize ideas into more complex interpretations and relationships than was reported nationally. In 2010, Shepherd’s reported results were comparable to all NSSE comparison groups.

6.  In 2005, Shepherd seniors were less likely (S) than BG and nationally surveyed seniors to apply theories in new situations or as a means to solve practical problems. In 2010, Shepherd’s reported results were comparable to all NSSE comparison groups.

7.  In 2005, first-year students at Shepherd reported being assigned fewer (M) course readings than peer and nationally tested respondents, and slightly fewer (S) readings than BG respondents. Shepherd seniors reported fewer (S) reading assignments than the comparison groups as well, but not by as wide a margin as   was reported by first-year students. In 2010, Shepherd’s reported results were comparable to all NSSE comparison groups.

8.  In 2005, first-year and senior students at Shepherd wrote fewer (S) papers in general than students nationally, at other BG schools, and at PG institutions, and reported submitting only between 1 and 4 papers of medium length (5-19 pages), while students elsewhere reported submitting between 4 and 8 papers of that   length, indicating a moderately (M) significant difference.  In 2010, only first-year students report writing fewer (S) papers of medium length (5-19 pages) than first-year students nationally or at other CC institutions.

9.  In 2005, Shepherd seniors are somewhat less likely (S) than BG and overall respondents to report that they have taken challenging exams. First-year students at Shepherd reported being challenged by exams on an average par with other first-year respondents. In  2010, Shepherd seniors are somewhat less likely (S)   than seniors at other SP schools to report that they have taken challenging exams, although their responses were closely in line with those of seniors at other CC institutions and nationwide. First-year students at Shepherd reported being challenged by exams on an average par with other first-year NSSE respondents.

10.  In 2005, first-year and senior Shepherd students were somewhat less likely (S) to exercise or participate in physical fitness activities than students in the three comparative groups.  In 2010, Shepherd’s reported results were comparable to all NSSE comparison groups.

 11.  In 2005, seniors at Shepherd were less likely (S) than PG, BG or national respondents to travel and study abroad.  In 2010, Shepherd’s reported results were comparable to all NSSE comparison groups.

12.  In 2005, both first-year students and seniors at Shepherd were less likely (M) than students from the PG and NSSE overall groups to have received or anticipate receiving adequate foreign language coursework. Both were also less likely (S)than BG respondents to have taken foreign language coursework.  In 2010, both   first-year students and seniors at Shepherd are still less likely (S) than students from the three comparison groups (SP, CC and NSSE overall) to have received or anticipate receiving adequate foreign language coursework.  However, the difference is less significant in 2010.

 13.  In 2005, first-year students at Shepherd were less likely (S) than students nationally or at other Baccalaureate/General Education institutions to be involved in community service or volunteer work. Seniors at our institution appeared to be as likely as students in the three comparison groups to volunteer or perform community service.  In 2010, Shepherd’s reported results were comparable to all NSSE comparison groups.

 14.  In 2005, first-year students at Shepherd perceived their relationships with faculty to be slightly less supportive (S) than students at other BG institutions. Seniors at Shepherd found the quality of their relationships with faculty members, other students, and administrative personnel, to be somewhat less supportive (S) than seniors nationally or at other BG schools.  In 2010, only seniors at Shepherd perceived their relationships with other students and faculty to be slightly less supportive (S) than students at other CC institutions, but responses were not dissimilar from those of students at other SP or nationwide institutions. 

15.  In 2005, Shepherd first-year students reported working for pay off campus exceedingly more (L) than respondents in the Peer Group, significantly more (M) than the national norm, and slightly more (S) than BG respondents. Seniors at Shepherd also were reportedly more likely (M) to require off campus jobs than seniors at PG schools, and slightly more (S) than the national norm.  In 2010,Shepherd’s reported results were comparable to all NSSE comparison groups.

16.  In 2005, Shepherd first-year students report that they must provide care fordependents living with them (parents, children, spouse, etc.) at a much higher rate (L) than PG respondents indicated, at a moderately higher rate (M) than was the national norm, and at a somewhat higher rate (S) than BG first-year students.  Seniors were also much more likely (M) than PG and somewhat more likely (S) than national respondents to be caring for dependents.  In 2010, Shepherd’s reported results were comparable to all NSSE comparison groups.

17.  In 2005, both seniors and first-year respondents at Shepherd reported a greater(S) amount of time spent commuting to class than students at comparativeinstitutions.  In 2010, Shepherd’s reported results were comparable to all NSSEcomparison groups.

18.  In 2005, Shepherd seniors indicated they did not feel their Shepherd education had significantly contributed to their spiritual development or to their development of a personal code of values and ethics. (Moderately less satisfied than BG or as reported by seniors nationally).  In 2010, Shepherd seniors reported   being only slightly less satisfied (S) in this regard.

19.  In 2005, both senior and first-year Shepherd respondents were slightly less satisfied (S) with the quality of advisement they received than was reported by students nationally or at other BG institutions.  In 2010, only seniors were less satisfied (S) with the quality of advisement they received than CC peers.


20.  In 2005, Shepherd seniors were moderately less satisfied (M) with their entire educational experience as compared with other seniors nationwide. In 2010, seniors at Shepherd were only slightly less satisfied (S) with their overall experience than respondents in the three comparison groups.

Challenges (gaps that matter) gleaned from the NSSE data gathered this past year include:

1.  In 2010, Shepherd seniors reported being less likely (S) to receive prompt written or oral feedback from faculty on academic performance than SP, CC or national norm.

2.  In 2005, Shepherd students reported being asked to complete comparable numbers of problem sets to their peers.  In 2010, however, results reveal that first-year Shepherd students feel they are asked to complete fewer (S) problem sets (both those that take less than an hour to complete and those that take more than   an hour to complete) than their peers at SP, CC & nationwide schools. Shepherd seniors also reported being asked to complete fewer (S) problem sets requiring more than an hour to complete than their peers in the three comparison groups.

3.  In 2010, seniors at Shepherd also felt the institution placed only some emphasis on encouraging contact among students from different backgrounds, a slightly lesser emphasis (S) than schools in the three comparison groups.

  4.  In 2005, first-year students at Shepherd felt that their institutional experience had contributed beneficially (S) to their understanding people of other racial and ethnic backgrounds in all three comparison groups.  In 2010, first-year students and seniors reported feeling less confident (S) than all three comparison groups in     this regard.

 5.  In 2010, FY & SR students felt that Shepherd had done less (S) to contribute to their use of computers and information technology than peers in the three other comparison groups (SP, CC, & NSSE).

6.  In 2010, FY & SR students reported feeling less confident (S) that their experience at Shepherd contributed to their ability to work effectively with others or learn effectively on one’s own.

THE STUDENT READINESS INVENTORY (SRI)

The SRI is an assessment designed to measure psychosocial factors that may influence student success in college such as social skills, motivation, emotions, hard work, commitment and determination.  Backed by sound research, the SRI increases an institution’s ability to detect at-risk students by as much as 50%.  The SRI can also recommend developmental interventions to assist individual students.  The test contains 108 questions and is administered via pencil and paper at the July Advisement/Registration sessions (and most recently in our required first-year seminar courses).  Individual student scores come to the Center for Teaching and Learning and are sent to each student and his or her advisor, along with a letter outlining success strategies, tools, and resources customized for each individual need.  The scores in no way impact student admission or placement and all individual scores are kept private. Preliminary analyses indicate that the SRI is, indeed a good predictor of student retention and success at Shepherd University. In Fall 2012, Shepherd will be piloting the newly revised SRI which is now called "Engage."

 

THE NOEL-LEVITZ STUDENT SATISFACTION INVENTORY  

The Noel-Levitz Student Satisfaction Inventory is administerd every three years at Shepherd University by Student Affairs. This survey asks students to rate how important various campus items are to them (i.e. parking, registration procedures) and how satisfied they are with current practices at the institution. Results can be used to do the following: guide strategic action planning, strengthen student retention initiatives,meet accreditation requirements, identify areas of strength for institutional marketing, and chart progress toward campus goals.  Our 2011 data shows remarkable improvement from our 2007 results.  We narrowed the gap between importance and satisfaction in many areas that were a concern in 2007 including the following items

Noel-Levitz questions that show improved results in 2010 vs. 2007

My academic advisor is concerned about my success as an individual.

Security staff respond quickly in emergencies.

There is a good variety of courses provided on this campus.

Faculty are fair and unbiased in their treatment of individual students.

I am able to experience intellectual growth here.

Adequate financial aid is available for most students.

It is an enjoyable experience to be a student on this campus.

Faculty are usually available after class and during office hours.

This institution shows concern for students as individuals.

The amount of student parking space on campus is adequate.

The campus staff are caring and helpful.

The personnel involved in registration are helpful.

Financial aid awards are announced to students in time to be helpful in college planning.

Students are made to feel welcome on this campus.

On the whole, the campus is well-maintained and safe and secure for all students.

Admissions staff are knowledgeable.

Freedom of expression is protected on campus.

I seldom get the "run-around" when seeking information on this campus.

Financial aid counselors are helpful.

Billing policies are reasonable.

Parking lots are well-lighted and secure.

My academic advisor helps me set goals to work toward.