Angelo Bummer
Integrating eLumen Technology
into the Student Learning Outcome Assessment & Analysis Process at a California Community College
This case study discuses the efforts of the Student Learning Outcomes Committee at Las Positas College to institutionalize a process for assessing student learning outcomes for all its programs as per the standards of the Accrediting Commission of Community and Junior Colleges (ACCJC). It describes the process of leveraging eLumen technology to achieve this goal and highlights the challenges of implementing a new, college-wide technology and getting every faculty member to use that technology. The SLO Committee encountered a culture of senior faculty resistant to SLO work, having not seen any value in the work previously, and a new technology that was not user friendly. The project was prompted by ACCJC recommendations during their 2015 review and began in 2016. While the process for implementing eLumen to facilitate SLO work across the college was completed in 2020, department efforts to collect accurate SLO data to inform program improvements are ongoing.
Executive Summary
Organizational Background
Las Positas College (LPC) is a community college in Livermore, California in the San Francisco Bay Area. Situated between the urban East Bay and the rural central valley, the college attracts a diverse mix of students. Las Positas College started as a satellite campus for Chabot College, located in nearby Hayward, in 1963, and it grew to establish itself as an independent college in 1988, receiving full accreditation in 1991. As part of the Western Association of Schools and Colleges, the Accrediting Commission for Community and Junior Colleges (ACCJC) oversees accreditation for Las Positas College. In 2015, the ACCJC performed a comprehensive review of the institution, which resulted in reaffirmed accreditation on condition that the college make progress on five critical recommendations focused on ACCJC standards. Among the list of recommendations was one that required the college to institutionalize a process to assess the student learning outcomes (SLOS) for all its programs. LPC’s Student Learning Outcome (SLO) Committee, comprised of faculty, staff, and administrators, was tasked with making progress on this recommendation. It has since made significant strides in institutionalizing a process. The college has followed up on all recommendations and reported its progress to the ACCJC with annual self-assessment reports and a mid-term report in Fall 2019. The college is due for the next comprehensive visit from the ACCJC team in Fall 2022.
Setting the Stage
College administrators, with the guidance of LPC’s Instructional Technology Coordinator, purchased eLumen technology in an effort to implement an institution-wide process to assess student learning outcomes. The task of the SLO Committee was to teach faculty how to use eLumen and guide each department to create and deploy a routine process for assessing student learning outcomes that is in line with the standards set forth by the ACCJC. This process began with teaching faculty how to create student learning outcomes in the first place. Many faculty were confused between course objectives, which are the discreet skills facilitated in the course, and student learning outcomes, which articulate the broad-based learning goals of the course. Most of the student learning outcomes already created for courses did not meet ACCJC standards, but they were nonetheless uploaded into eLumen. First and foremost, SLOs needed to be revised and entered again into the system. The reason there were so many poor quality SLOs in the first place was due to the fact that faculty had been required in the past to create SLOs in a short amount of time. Most faculty had never written SLOs before, and they were prompted to do so without training. This caused many faculty to view creating SLOs as pointless “busy work” and contributed to a culture, particularly among senior faculty, of resistance to seeing the value in the process.
Another challenge was learning how to use eLumen. The members of the SLO Committee did not receive any formal training on how to use the program, but they learned from each other on how to use several features that were most important to the task of inputting and assessing SLOs, and they quickly realized that the system would not be intuitive for other faculty. This presented two complications. First, the SLO Committee needed to simplify the user experience so that it would be less confusing for faculty to use. Second, the SLO Committee needed to adjust their plan of action when they learned about new features or limitations of the eLumen system. Finally, the SLO Committee, knowing the long-term goals of the project, knew they needed to roll out the plan in phases, while also offering support to departments that implemented each phase at a faster place. The Committee needed to (1) create accurate SLOs for each course, (2) input and assess those SLOs in eLumen, (3) analyze and leverage SLO data as a department to inform program decisions that would increase student success, (4) create and follow a plan to ensure each SLO is assessed within a three-year period as per ACCJC standards, and (5) organize and plan routine meetings to analyze SLO data. An additional and overarching concern was the fear that since faculty, both full and part time, were extremely busy, adding an additional responsibility to their schedule might have limited success.
Case Description
Project Definition: To implement eLumen technology as a means to facilitate a college-wide process of collecting accurate student learning outcome data and perform meaningful analysis of that data to inform program decisions.
Planning
To plan for the project, the SLO Committee knew they needed to advocate for monetary compensation to incentivize faculty to complete their SLO work. To do this, the SLO Committee met with the Faculty Association (FA), which was in the process of negotiating a new faculty contract for the district (this happens every three years). Salary is always a negotiating point, and in the end, the FA was able to secure wage increases for several reasons, including making progress on implementing and assessing SLOs. Part-time faculty were also given a raise in conjunction with a new work requirement to complete a certain number of “professional responsibility hours” per semester, SLO work included. This was an initial step to get buy-in from faculty to devote time to learning eLumen and complete SLO work.
Faculty would also need support. To do this, members of the SLO Committee (1) created workshops focused on creating and assessing SLOs in eLumen, and (2) negotiated with administrators to fund “SLO Liaison” positions. One SLO Liaison was allotted for each division and would serve as a point-person to faculty, providing one-on-one assistance with crafting student learning outcomes and using eLumen to assess them. Given budget constraints, SLO Liaisons were allotted one unit of reassigned time per semester for their work. I joined the SLO Committee as a SLO Liaison for the Arts & Humanities division, so I did not play a role in selecting eLumen as the institutional technology to facilitate SLO assessment nor did I play a role in advocating for salary raises as a means to incentive faculty to devote time to creating and assessing SLOs.
Implementation
As part of the support team, a critical resource to offer SLO and eLumen workshops was “Flex Day,” a full day of professional development workshops in which faculty are required to participate once a semester. Typically, one facilitator would guide students through the process of crafting SLOs and/or assessing SLOs in eLumen and a team of support staff would provide individual help. These workshops were useful as were the SLO Liaisons particularly in the first phase of implementation, that of creating the SLOs in line with ACCJC standards. The SLO Committee also reviewed each new SLO inputted into eLumen to ensure quality by making recommendations as needed.
While it was also valuable to teach faculty how to create SLO assessments in eLumen and input SLO data, a lack of understanding of how eLumen functioned caused us to alter our approach. It turns out that when faculty create SLO assessments in eLumen, the SLO data associated with them does not map to program-level learning outcomes and institutional-level learning outcomes. That data is stored on its own for that particular course; it cannot be viewed, for example, in conjunction with all the other sections for the course, a necessary function to analyze SLO data to inform program changes, which is the point of the project. Only when faculty assigned in the eLumen coordinator role for their department create the SLO assessments does the SLO data map to program and institutional outcomes. Needless to say, the SLO Committee altered its approach to train department coordinators to create SLO assessments in eLumen.
In designing the coordinator-specific workshop, the SLO Committee decided to create templates for SLO assessments, and it revised the SLO reflection template in an effort to simplify the process of creating SLO assessments and assessing SLOs in eLumen and make the user experience in eLumen more intuitive. It was at this time, however, that funding was cut for the SLO Liaisons, so the SLO Committee relied heavily on facilitating coordinator workshops during “Flex Day” to implement the changes. The process, however, was a lot quicker and straight forward. The templates saved coordinators a lot of time inputting the assessments. Also part of this training was how to assign faculty SLOs to assess for their courses.
Overall, the process of assisting faculty to craft accurate SLOs for their classes and training coordinators to create assessments and assign SLOs to faculty took over three years. Given the removal of “SLO Liaisons,” department coordinators were designated to assist faculty in using eLumen, but this task was no longer burdensome since the assessments and reflection templates simplified the user experience in eLumen.
The next phase of the project was to align faculty and departments with the frequency of assessment and analysis as outlined by ACCJC standards, which states that all SLOs must be assessed a minimum of once every three years with meaningful analysis of SLO data to improve programs taking place. To assist in facilitating this, the SLO Committee created a 3-year assessment plan template, which assisted coordinators in planning which SLOs to assess and when, as well as planning annual SLO data analysis meetings. Coordinators completed these and returned a copy to house on Student Learning Outcomes page on the college website. In addition, an SLO section was revised in the program review report that departments are required to complete each year; in the SLO section, departments report out on their progress with collecting SLO data and leveraging that data to inform decisions to improve student success.
While the process is in place for assessing student learning outcomes, and most coordinators have been following through with setting up assessments for faculty on a semester-to-semester basis, many departments still struggle with getting enough faculty to input their SLO data in eLumen. Assistance from the SLO Committee is ongoing, and the number of course sections assessed are increasing. For example, within the English department, about fifty percent of faculty are completing their SLO work routinely. Other departments have more or less success, but this continues to remain an obstacle, and while the structures are in place for institutionalized assessment of SLOs, integrating robust assessment and regular analysis in a routine and meaningful way can be a slow and varied process for departments.
Assessment
As noted above, a lack of sufficient understanding of how eLumen technology functions stalled the project, and the effort to simplify the eLumen user experience allowed for much more efficiency. Another thing that stalled the project was pressures from administrators to increase the number of SLOs assessed in eLumen; they wanted to include an eLumen report in their Self Evaluation report to the ACCJC under the assumption that higher numbers meant progress. The point of the project is not only to institutionalize a process for SLO assessment, but to more pointedly enable departments to have meaningful conversations and make data-driven decisions with regards to improving student success with learning outcomes. In this sense, an initially poorly defined project contributed to the stall in the project. The rush to increase the number of SLOs assessed in eLumen occurred during at the same time as the SLO Committee was trying to train faculty to create assessments and input SLO data in eLumen. As we now know, this turned out to be a waste of time. This experience also likely reinforced the impression of SLO work as “busy work” to faculty who were resistant to devoting time to it, and possibly others who felt their time was wasted. The adjustment to department coordinator controlled SLO assessment, the use of templates to simplify the eLumen user experience, and the shift in focus for departments to work towards a process of meaningful SLO analysis were critical to restoring a sense of value in the SLO assessment process and creating more buy-in among faculty. As it stands right now, departments are at varied phases in this process but most are focused on collecting sufficient and accurate SLO data so that department discussions and analysis of that SLO data can be meaningful and lead to effective data-driven decisions.