Date: Thu, 19 Aug 2004 08:19:53 -0400 Reply-To: ASSESS - Assessment in Higher Education <[log in to unmask]> Sender: ASSESS - Assessment in Higher Education <[log in to unmask]> From: "Suskie, Linda" <[log in to unmask]> Subject: Re: pre- post testing in assessment Content-Type: text/plain; charset="utf-8" People who believe pre/post designs are the only way to go touch a nerve with me because, like any other approach to assessment, this approach has both pros and cons, which should be weighed against the pros and cons of other possible approaches. As others have noted, the #1 consideration should be matching one's approach to one's goals, and most colleges I know are less concerned with individual growth than simply whether students are meeting appropriate standards when they pass a course or graduate: Are they writing at the level we want and expect for a college graduate? Analyzing and evaluating at an appropriate level? etc. etc. In these cases, as others have noted, the post-only approach is appropriate; the big challenge is determining an acceptable competence standard, especially for locally-developed instruments. Using a pre/post design in these situations simply needlessly doubles one's assessment workload. But there are other reasons to think twice about pre/post approaches: 1. They're simply unnecessary for courses and programs in which students enter with no prior knowledge or skills--occupational therapy, for example. 2. It can be very hard to motivate students to do their best on the pre-test--it shouldn't be graded, of course, since the students haven't been taught anything yet, so what's to stop a student from scribbling meaningless words on a paper or filling in all the B bubbles on a test? 3. Pre-tests may not be available for students who transfer into an institution or program (i.e., change majors), a particular problem at institutions with large numbers of transfer students. 4. If we do indeed see a significant gain, we often can't be sure it's due to our courses/program and not to other experiences or normal maturation. A student might have a concurrent part-time job, for example, that has improved her oral communication skills far more than her required speech course. 5. Pre/post results seem harder for faculty to interpret than post-only results. Suppose that, on a 5-point scale, your students average 3.6 pre-test and 3.9 post-test. Even if this is a statistically significant difference, is it as big a difference as we hope for? Figuring out how much growth is appropriate/expected can be very difficult. 6. (and this is my biggest concern) Pre/post designs can be extremely imprecise (i.e., full of measurement error). If you've taken statistics, you know that the standard error for the difference between two means is roughly double the standard error for one of those means. On top of that, many of our locally-developed assessments (rubrics, tests, etc.) can be highly imprecise--faculty may not give the same essay the same score consistently, and providing appropriate training and practice and having faculty read every essay twice to calculate interrater reliabilities is possible but doubles time and cost. So all this measurement error can easily mask significant gains. For what it's worth, there are other perspectives (I call them frames of reference) that can be useful in understanding assessment results, depending of course on what your goals are. If your goal is that your students surpass their peers, it's appropriate to compare them against some kind of norms. If your goal is that students be among the best among their peers, a "best-practice" frame of reference is appropriate. If your goal is that you do a better job teaching than you did a few years ago, a longitudinal perspective is appropriate. For more information on all these frames of reference, see Chapter 6 of Assessing Student Learning: A Common Sense Guide (Anker, 2004). Linda Linda Suskie, Director Office of Assessment Towson University 8000 York Road Towson MD 21252 Voice (office): 410-704-2620 Voice (home): 717-898-9247 Fax: 410-704-6330 E-mail: [log in to unmask] <mailto:[log in to unmask]du> Web: www.towson.edu/assessment <http://www.towson.edu/assessment> -----Original Message----- From: ASSESS - Assessment in Higher Education on behalf of Peter J. Simon Sent: Wed 8/18/2004 1:42 PM To: AS[log in to unmask] Cc: Subject: Re: pre- post testing in assessment I agree that an assessment program must have measurable goals. And multiple measures are important. But that’s part of my problem. Everyone here wants to start and finish with pre and post tests. I looked up pre and post testing for the assessment of student learning on the ASSESS archives and did not find anything since August 2000. I may try to find the paper Dennis wrote. His synopsis says it all though, even if he shows that it’s not “necessarily the case.” But no one I know ever grades on the basis of improvement anyway—and, as Dennis’s synopsis suggests, for good reason. I believe that we’re getting pressure from our Accreditation agency, but I think that the reason we’re heading toward pre and post testing is that it will almost always show student learning—and so it “proves” that we’re doing what we say we’re doing. I would like to start here with Erin’s suggestion that we start with some post “tests” at this point and then go on to pre and post tests later perhaps: it won’t be easy to convince the group. But my main concern is still competency: pre and post tests cannot determine how well something is learned or how much relative to what the course offered or any level of proficiency—at least not easily. I proposed that we give some sort of final, and then with a bit of smoke and mirrors with some statistical techniques show that at some predetermined level of significance, say 95 percent, that if we pick out a student at random and give him or her the test he or she would pass the test 95 percent of the time—with a sufficient sample size. If the final exam was a fair and accurate measure of the course then I think that that would be a good (not perfect) indicator of student learning—and faculty teaching as well perhaps. Thanks for your help and suggestions everyone, Peter _____ From: ASSESS - Assessment in Higher Education [mailto:[log in to unmask]] On Behalf Of Mark A Davenport MADAVENP Sent: Wednesday, August 18, 2004 10:03 AM To: [log in to unmask] Subject: Re: pre- post testing in assessment Dennis makes an excellent point. A point that only reinforces the need for multiple methods and multiple measures. Much of this comes down to how you choose to demonstrate learning. If you SAY that you are going to document changes in student learning then you had better be prepared to provide evidence of change. As Dennis and others have noted, this isn't easy to do. However, if you say that is what you are going to do, your accreditor is going to expect to see the evidence and it will not accept satisfaction or self-report evidence. We recently replied to a SACS recommendation regarding our assessment program and got a very good response back. The core of our change was very simple: Carefully review our mission and our goals, make sure everything is measurable, and made sure we had (or could develop) multiple sources of data to document outcomes associated with each goal. I have found out that a big part of our problem was language. In Student Affairs (and in Academic Affairs), we know why we are here and we do believe that students learn, change, develop across many dimensions. Problem is, most have rarely if ever been required to actually, objectively document this learning, change and/or development. For our part, SACS won't let us operate that way anymore. I now review all of our departmental goals for measurability (Note: I didn't say 'objective' measurability. Any valid source is helpful). If we can't measure it, we rewrite it so we can measure it. Is this a cop-out? Are we 'teaching to the test' so to speak by formenting the impression that we alter our outcomes to satisfy accreditiors? No. We simply make sure that we write our goals in terms of outcomes that we know we can measure. Mark *************************************************************************************************************************************************************** Mark A. Davenport Ph.D. Asst. to the Vice Chancellor for Student Affairs Office of Student Affairs Research and Evaluation The University of North Carolina at Greensboro 336.344.5582 M_Davenport@uncg.edu 'An approximate answer to the right question is worth a good deal more than an exact answer to an approximate question.' -- J. W. Tukey ******************** Information on other assessment-related sites can be found at http://www2.acs.ncsu.edu/UPA/assmt/resource.htm The oral history of ASSESS and instructions for subscribing and unsubscribing are available at http://lsv.uky.edu/archives/assess.html ******************** Information on other assessment-related sites can be found at http://www2.acs.ncsu.edu/UPA/assmt/resource.htm The oral history of ASSESS and instructions for subscribing and unsubscribing are available at http://lsv.uky.edu/archives/assess.html ******************** Information on other assessment-related sites can be found at http://www2.acs.ncsu.edu/UPA/assmt/resource.htm The oral history of ASSESS and instructions for subscribing and unsubscribing are available at http://lsv.uky.edu/archives/assess.html Back to: Top of message | Previous page | Main ASSESS page

LSV.UKY.EDU

University of Kentucky Computing Center 128 McVey Hall Lexington, Kentucky 40506 859 257-2900 (voice) 859 323-1978 (fax) listmaster@lsv.uky.edu (e-mail)