SF State Writing Program Assessment Page
Assessing Student Learning with an Equity Lens
Over the last five years, we have collected and analyzed 2000+ pieces of first-year student writing. We believe that this is the largest archive of student writing ever collected on our campus. And we believe it has much to tell us about how students experience their first year of college and develop as college writers. As importantly, with this archive, we've taken an approach to assessment that is asset-based and improvement-oriented. Such an approach helps us build richer understandings of students' literacy experiences and support principles of antiracism and linguistic justice.
Our method: Beginning with teacher expertise and student writing, we partnered with Writing Program faculty to identify core curricular values and to design a reflection-centered assignment for program assessment. Over the next several years, we invited teachers to interpret student learning via those assignments in an assessment community of practice (CoP). We crafted SF State Writing Program-specific threshold concepts that cut across our curricula, and built learning maps for each concept. Each learning map centers what students told us in their reflections about their experiences and learning as writers, including what West-Puckett, Caswell and Banks call “sideways paths” -- the detours and seemingly unrelated stops that learners often take.
Learning maps thus consist primarily of a synthesis of students’ own descriptions of their learning trajectories, married with teachers’ descriptions of typical students’ progress. In our learning maps, we focus on understandings (Adler-Kassner and Wardle) rather than judgments about the quality of students’ writing or their “mastery” of outcomes (Gallagher). What have we discovered in our assessment so far? Among many findings related to first-year retention and writing development (see reports below), we are uncovering how students think about their own writing, as well as the affective dimensions of their experiences as writers in school.
Click on one of our seven threshold concepts to reveal the learning map for each concept, which describes students’ learning from emerging to meta-understanding. The learning maps document students’ possible progress across their entire undergraduate journeys, with students attaining meta-understandings in or even after their GWAR class. Maps are meant to be descriptive and non-linear, allowing for the sideways pathways evoked by the image at the top of this page.
-
Emerging Understanding
-
Has not been given opportunities to bring voice, self, experience, or identity to their writing in school (e.g. has been taught to avoid using “I”; has been taught to focus on performing tasks over making meaning; has not been given opportunities to connect writing to their own goals).
-
-
Developing Understanding
-
Recognizes that they can use their voice, self, experience, or identity in their writing in school (e.g. uses “I” in personal or reflective writing, or to express understanding or explore difficulties. Has been taught that some assignments allow for “creativity” but most do not).
-
-
Moving Toward Understanding
-
Understands the importance of bringing voice, self, experience, or identity to their academic writing and uses this connection to engage or direct their education (e.g., connects self to inquiry topics; uses writing purposefully to gain self-knowledge).
-
-
Understanding
-
Understands and can analyze the importance of bringing voice, self, experience, or identity to their academic writing (e.g. not only expresses their own interest/connection to topic, but draws the reader in as well; makes and analyzes connections between self and writing/topic).
-
-
Meta-understanding:
-
Articulates how they take an active role in using writing to connect aspects of their selves/lives; situates their identity in the discourse around the topic (e.g, connects lived experiences, interests and reflection to the topic and the audience).
-
-
Emerging Understanding
-
Has been taught to see writing acontextually and/or to view judgments about school writing as either “objective” or purely “subjective” (e.g. has been taught that individual teachers or authorities determine writing quality; has experienced writing in school as a requirement to fulfill in order to please the teacher and get a good grade). Thus, does not yet realize the impact of identities, cultures, or power structures on writing.
-
-
Developing Understanding
-
Recognizes that there is more than one way to write; recognizes that judgments about “good” writing or other language use are complex and tied to identity and culture.
-
-
Moving Toward Understanding
-
Understands and begins to articulate how writing, culture, and identity are related and impact one another (e.g. understands that judgements around writing are shaped by power and culture).
-
-
Understanding
-
Analyzes or critically examines connections between writing, culture, and identity. Understands that writing can be a form of action and/or power; analyzes assumptions about language difference/diversity.
-
-
Meta-understanding
-
Articulates how they use connections between writing, culture, and identity to accomplish rhetorical goals (e.g. calls readers in; can appeal to multiple discourse communities and/or use multiple linguistic approaches and modalities to accomplish rhetorical goals).
-
-
Emerging Understanding
-
Has not been given opportunities to use processes of drafting, feedback, collaboration, or revision (e.g. and thus may struggle to bring ideas to the page; brainstorming and drafting are difficult; may write too little). Revision is limited, or school writing experiences may have drained the needed confidence to make choices without direct teacher instruction.
-
-
Developing Understanding
-
Recognizes the importance of drafting, feedback, collaboration, and revision (eg. can generate and brainstorm lots of ideas, but may still revise only when prompted or write shorty before due date).
-
-
Moving Toward Understanding
-
Understands and articulates the relationship between writing processes and successful writing (e.g. may use various strategies for drafting. Reflects on/uses feedback from teachers and peers; notices that writing improves with revision).
-
-
Understanding
-
Analyzes how writing processes influence final result; analyzes feedback and revises actively (e.g. revises globally as well as locally, focusing on sharpening/clarifying ideas/purpose as well as improving organization and sentences).
-
-
Meta-understanding
-
Articulates how/shows evidence of how feedback and revision have improved / changed their writing and thinking.
-
-
Emerging Understanding
-
Has been taught to see audience, purpose, and genre in unidimensional terms or solely in terms of the teacher/assignment/grade (e.g. and thus writes for the teacher or grade). Relies on school formulas, formats, or prompts. Believes there is one way to write.
-
-
Developing Understanding
-
Recognizes that there are many different types of writing, and audiences may vary in their response; recognizes that writing should have a purpose; recognizes that there is not one correct way to write.
-
-
Moving Toward Understanding
-
Understands that audience, purpose and genre are context-specific and complex. Budding awareness that genres both constrain and enable meaning and form.
-
-
Understanding
-
Articulates how they analyze aspects of audience, genre, and purpose in order to achieve rhetorical goals (e.g. may experiment with new genres; uses understanding of genre to attempt to reach audiences beyond the teacher).
-
-
Meta-understanding
-
Articulates how they use understanding of audience, genre and purpose to achieve rhetorical goals (e.g Is able to transform genres to suit their own purpose. Addresses professional/disciplinary audiences).
-
-
Emerging Understanding
-
Has been taught to rely on teachers for evaluation or reflection rather than self or peers. Has had limited opportunities to see how academic habits of mind support learning, reading, and writing.
-
-
Developing Understanding
-
Begins reflecting and thinking about their own thinking; budding awareness of how academic habits of mind support learning, reading, and writing.
-
-
Moving Toward Understanding
-
Explores and practices reflection which improves thinking and writing; understands and can self-evaluate their own habits of mind and where they need to grow in terms of writing.
-
-
Understanding
-
Reflects (and/or articulates how they reflected) on their own thinking, reading, and writing; questions their own thinking. Expresses curiosity, risk-taking, openness. Consistently uses habits of mind to aid and deepen learning.
-
-
Meta-understanding
-
Can critically analyze how they use metacognition and habits of mind to accomplish writing goals. Actively uses habits of mind to improve writing.
-
-
Emerging Understanding
-
Has been taught to read transactionally for “knowledge reporting” or “quote-mining,” and thus may not yet incorporate readings or other texts in their own writing.
-
-
Developing Understanding
-
Draws on readings that have impacted their thinking. Raises substantive questions about texts.
-
-
Moving Toward Understanding
-
Explores and practices how reading involves knowledge-transformation, rereading, and attention to contexts. Grapples with difficulty even when not prompted.
-
-
Understanding
-
Articulates how they transform knowledge gained from reading; uses questions to deepen analysis of texts; considers context to deepen analysis.
-
-
Meta-understanding
-
Reads critically and analytically; reflects on how reading and re-reading affect understanding.
-
-
Emerging Understanding
-
Has been taught a “checklist” methodology to find and evaluate sources or may only use research when prompted; may not know how to find and use sources; views sources in the context of supporting existing opinions. Over-trusts outputs from technology; believes machine-generated writing is more “correct” and “better” than their own.
-
-
Developing Understanding
-
Budding awareness that evaluating sources is necessary but relies primarily on checklists. Recognizes the importance of citation practices; attempts citation even if it is not yet correct. Begins to consider sources that complicate their own opinion.
-
-
Moving Toward Understanding
-
Practices using sources and can use them effectively in their writing. Explores how and why we evaluate and cite sources to generate and support ideas or arguments. Begins to use sources to broaden or deepen their perspective or understanding.
-
-
Understanding
-
Articulates how and why they analyze and interrogate sources; considers and situates credibility of sources. Uses various search strategies to locate sources; cites sources ethically. Uses sources to broaden and deepen their own and readers’ understanding.
-
-
Meta-understanding
-
Critically analyzes the complexity involved in their analysis of sources, including different kinds of sources based on purpose. Can analyze the assumptions and power relations involved in knowledge-creation and presentation.
-
Our Scholarship
-
Personal, Accessible, Responsive, and Strategic Assessment: Creating a Faculty Community of Practice (Trainor and Holland, 2024)
-
Redesigning Graduate Composition Courses for Justice: A Case Model for Promoting Access, Inclusivity, and Trauma-Informed Pedagogy (Gelms, Gilligan, Kohls, Lockhart, and Roberge, 2021)
-
The Critical Place of Reading in Writing Transfer (and Beyond) (Lockhart and Soliday, 2016)
-
Rethinking Regulation in the Age of the Literacy Machine (Soliday and Trainor, 2016)
-
Critiquing the Need to Eliminate Remediation: Lessons from San Francisco State (Goen-Salter, 2008)
Learn More about our Assessment Study
Rubrics list criteria that are often framed in deficit language ("thorough; adequate; minimal; basic"). They locate failure in teachers and students rather than in institutions and systems. They lead readers to judge rather than interpret students' writing and learning. And they too often focus our attention on end goals, obscuring the learning that happens along the way.
Unlike rubrics, our learning maps encourage teachers to use thick descriptions to analyze students’ writing. They help us avoid the white language supremacy that is often encoded in traditional assessment materials, and enable us instead to take a rhetorical and critical approach -- for example, we have a threshold concept focused on language, culture and power, rather than the more traditional learning outcome focused on grammatical correctness.
To learn more about how rubrics (especially generalized rubrics) are often steeped in colonizing ideology that privileges White Mainstream English (WME) while further marginalizing the voices of students, we recommend the work of linguistic scholar April Baker-Bell, as well as scholarship on decolonizing assessment (see this MSU guide).
- Overview of the Writing Program’s self-study, undertaken in 2019-20 and 2020-21: describes a sea change in the Writing Program, instigated by executive orders in 2017 and our campus’ student success initiatives. This sea change involved substantive curricular revision and changes to the program’s culture, and provides a backdrop for our self-study analysis.
- Focus on student success, providing a preliminary analysis of DFW rates and student experiences as represented in SETE comments and scores
- Summarize our self-study, conducted by 12 lecturers and three tenure-track faculty.
- Describe our methods, including dynamic criteria mapping (Broad, 2003), curricular analysis, and team scoring of student writing, all with a focus on “closing the loop”
- Discusses key findings
- AY 2021-2022 A2 Writing Program Assessment Report: describes year three of our self-study project, with a focus on teacher collaboration, learning, and growth; struggling students and equity gaps; and, student barriers to learning and self-efficacy
- Read and analyzed 590 student reflections
- Logged approximately 45 faculty development hours with our assessment team, and created three workshops addressing needed improvement that reached over 60% of our faculty, and documented a marked increase in student facility with writing processes after last year’s interventions
- Significant finding from this yearʼs analysis: stretch (104-105 and 201-202) students were rated as further along in their journey toward understanding of program threshold concepts (see Adler-Kassner and Wardle, 2016) than accelerated A2 (114 and 209) students. In addition, there were fewer struggling students in stretch A2 (only 5% were rated as “dependent”, in comparison to 23% in 114) by the end of the course term.
- Beta-tested a self-efficacy question in A2 and AE - initial results show an approximate 2 point gain (on a 10 point scale) in students confidence and self-efficacy at the conclusion of their classes
- AY 2022-23 CEETL Report
- CEETL funding allowed assessment to continue with this year’s focus on anti-racist assessment and mapping of student learning, combined with in-depth faculty development.
- Preliminary analysis conducted and completed of 20 equity-based interviews (one-on-one qualitative interviews with faculty)
- Analyzed DFWU patterns across the first year and described needed interventions
- Beneath the Data: Why Students Receive NCs in A2 (May 2023)
- Using an internal study of NCs, this report demonstrates that 100% of the No Credits given in the Writing Program are actually unauthorized withdrawals.
- LCA data reveals that 80% of the students who unofficially withdraw from our A2 sections also receive no credits in all their other classes.
- This data also confirms the success of the program’s adoption of anti-racist assessment practices: over 80% of faculty in our program use some form of contract or holistic grading practices that don’t penalize students for linguistic diversity, and don’t grade based on white / middle-class discourse norms. As a result, no student in the Writing Program fails A2 due to individual teachers’ judgements about the quality of their writing.
- This report also provides data about when/why students disappear from A2 classes, patterns discerned from analyzing that data, and thick description of teacher interventions/attempts to invite students back to the work of the class.
- July 2023 A2 Assessment Report AY 2022-23_Equity Focus: documents equity-based research, data analysis using a data justice framework, and teacher-driven, antiracist assessment efforts aimed at lowering DFW rates and closing equity gaps ($12k of SSGI Equity 5 funding)
- Significant gains in A2 students’ confidence and self-efficacy
- Consistent gains in student learning, particularly attainment of key threshold concepts related to both literacy development and belonging and inclusion;
- Provides a more nuanced picture of students who DFW (e.g., nearly all DFWs in A2 are withdrawals that students did not submit; 80% of students who DFW in A2 also receive DFWs in the rest of their classes)
- Significant benefits of the Writing Program’s unique approach to program assessment: e.g., combining paid assessment work with professional development leads to powerful changes in teacher practice