Understanding Survey Results
Excluded Respondents
Respondents Included in Raw Data File
Benchmark Reports
Means and Frequency Reports
Standards for Interpreting Mean Differences
Weights and Local Student Characteristics
Student Level Breakout Definitions
Student Identifier Data
College leaders will wish to familiarize themselves with Survey of Entering Student Engagement (SENSE) findings before communicating about the results. The following notes should be considered.
Excluded Respondents
The total count of respondents in an institution's raw data file will differ from the numbers reported in the institutional reports due to intentional exclusion of certain surveys in the online reporting system. Respondents are excluded from the institutional reports for the following reasons; however, no students under the age of 18 are included in the raw data file or institutional reports.
- The respondent did not indicate full-time or part-time enrollment status at the institution. (Variable: ENRLMENT is blank)
- The respondent did not indicate entering or returning student status. (Variable: TERMSENR is blank)
- The survey is invalid. A survey is invalid if a respondent answered all items in number 19 as either never or four or more times. (Variables: ASKQUES, PREPDRFT, LATETURN, NOTTURN, SUPINSTR, NOTCOMPL, PINCLASS, PREPOUTC, GRPSTUDY, NRGSTUDY, USEINTMG, MAILFAC, FACASSN, CLASSREL, FEEDBACK, RCVGRDS, FACIDOC, OCIDEAS, SKIPCL)
- The respondent indicated their age as under 18. Note: Data for these students are not returned. (Variable: NEWAGE is 1. Responses where age was not marked are included in the analysis.)
- The respondent indicated they had taken the survey in a previous class or did not respond to item 1. (Variable: SRVAGAIN is 1)
- Oversampled respondents are not included because they are selected outside of SENSE's primary sampling procedures. (Variable: PSAMPLE is 0)
- The respondent indicated that they were a returning student (Variable: TERMSENR is not 1)
Respondents Included in Raw Data File
Raw data files contain responses from all survey instruments returned by the college, with the exception of invalid surveys and those completed by students under the age of 18. For the purposes of working with your data file, excluded respondents do not have a weight listed for the IWEIGHT variable. To run analysis without excluded respondents, simply remove any observations where IWEIGHT is missing. This will ensure that the analysis only includes primary sample respondents who do not meet any of the exclusionary criteria.
For the online administration, the number of respondents contained in the raw data file may differ from the data displayed in the Responder Tool during the administration. The Responder Tool only displays the number of respondents that reach the end of the survey; whereas, the data file contains data from all responders except for those under the age of 18, those who did not report enrollment status, those who did not report entering or returning student status, and those who provided invalid responses on the 19 sub-items in item 19.
Benchmark Reports
SENSE benchmarks are groups of conceptually related survey items that focus on institutional practices and behaviors that promote engagement among entering students. Benchmarks are used to compare each institution's performance to that of similar institutions and with the SENSE Cohort. The six SENSE benchmarks are: early connections, high expectations and aspirations, clear academic plan and pathway, effective track to college readiness, engaged learning, and academic and social support network.
Benchmark reports consist of tables showing the college's scores on each benchmark, followed by means and frequency tables of items in each benchmark. Benchmark scores provide an overview of how the college is doing in particular areas. Colleges should examine results from the individual survey items composing each benchmark as well.
Benchmarks Over Time
If your college has participated in multiple SENSE administrations, you may be interested in exploring your raw benchmark scores over time. The raw benchmarks over time graph can be found in the Excel benchmark report file.
Means and Frequency Reports
Responses to individual SENSE survey items are summarized in two formats: means and frequencies.
Means reports present an average for each survey item that has scaled responses (e.g., strongly agree to strongly disagree) and compare average item responses between member colleges and various groups, or between student subgroups within a college. Means are not run on dichotomous items (those with only two response options). These items are summarized in frequency reports.
Frequency reports present the observed frequencies of occurrence (counts and percentages) of the values for each survey item (excluding demographic survey items). These reports are useful for understanding how data are distributed across response categories. Counts and percentages on frequency reports are subject to rounding.
Standards for Interpreting Mean Differences
When interpreting mean differences across comparison groups, CCCSE uses a combination of two measures: (1) a t-test with a very conservative alpha level of .001 or less (to determine if the difference between two means is significant and not likely due to chance) and (2) an effect size of .20 (absolute value) or more using Cohen’s d (to show the magnitude of difference between the two means). If a comparison meets both of these criteria, then it is considered to be a statistically significant difference worthy of further investigation. Comparisons that meet these criteria are marked with a double-asterisk (**). For internal analysis of small groups, it may make sense for colleges to use a larger alpha level but typically not a larger effect size.
SENSE Sampling, Weighting, and Local Student Characteristics
Paper-and-Pencil Administration
In SENSE sampling procedures, students are sampled at the classroom level. As a result, full-time students, who, by definition are enrolled in more classes than part-time students, are more likely to be sampled. This introduces a sampling bias resulting in disproportionately more full-time students completing the survey. Statistical weighting, discussed in detail below, is used to adjust for this bias.
Online Administration
In order to boost responses to the online survey, sampling is not employed; the survey invitation is sent to all eligible students. Research shows that full-time students and women are more likely to complete online surveys than part-time students and men. As a result, full-time students and women are over-represented in the final sample. Statistical weighting, discussed in detail below, is used to adjust for this bias.
Statistical Weighting
Beginning with SENSE 2022, CCCSE staff began using the IPEDS counts for degree/certificate-seeking first-time and degree/certificate-seeking transfer-in fall enrollment instead of total fall enrollment. This change should provide more accurate weights for the SENSE population. One additional change to weights for SENSE 2022 is based on research that has found that full-time students and women are more likely to complete online surveys than part-time students and men. The national distribution of survey respondents reflects this pattern. Therefore, weights are now based on both enrollment status and gender. These new weights are applied to both the paper-and-pencil and online survey results. The use of the new wights will not negatively impact analysis of the paper-and-pencil surveys. For example, if the proportion of men and women among survey respondents exactly matches the proportion in the institution population, results based on the new weights would be the same as results based on enrollment status alone. The weight variable in the download data set is still named IWEIGHT. When conducting analyses in which either enrollment status or gender is included as a breakout variable, weights should not be applied.
The online SENSE administration may result in different survey participation patterns by student characteristics (e.g., enrollment status, gender) compared to an in-class administration. Therefore, CCCSE staff encourages you to closely review the student characteristics of the survey respondents compared to the characteristics of your college’s entire student population. If there are substantial differences between the two groups, calculating weights that specifically take into account these differences may be useful in analyzing your raw data file. CCCSE’s research team is available to assist with this task. Contact data@cccse.org for guidance and/or assistance.
Deactivating Weights
Under certain circumstances, deactivating weights may be a more informative way to examine institutional SENSE data. Even the most recent IPEDS data are approximately two years old and may not always accurately represent a college's current student population. For example, in the case that a college has experienced a significant change in enrollment characteristics during the two years prior to administering SENSE, the college's institutional research department may want to consider whether the weights based on the IPEDS numbers are completely appropriate.
Please note that with the weights now based on both enrollment status and gender, the decision to deactiveate weights becomes a little more complicated. The decision to deactivate weights in the online system should be based on the distributions of both enrollment status and gender. For example, if the institution population is 92% men and 92% full-time, then you may want to look at unweighted results. However, if the institutional distribution of either variable is not dominated by one enrollment status or gender subgroup, then the decision to deactive weights should be carefully considered.
Student Level Breakout Definitions
Breakout reports, including benchmarks, means, and frequencies, for institutional results are available in each of the areas below. Each category is based on student responses to specific survey items.
Full-Time & Part-Time (Enrollment Status)
Item 2: “Thinking about this semester/quarter, how would you describe your enrollment at this college?”
Developmental & Non-Developmental
The first three sub-items in Item 17: “In which of the following types of courses were you
enrolled during your first semester/quarter at this college?”
17a. Developmental Reading (also referred to as Basic Skills, College Prep, etc.)
17b. Developmental Writing (also referred to as Basic Skills, College Prep, etc.)
17c. Developmental Math (also referred to as Basic Skills, College Prep, etc.)
If a student responded they were enrolled in one or more of these types of courses during their first semester/quarter at the college, they are classified as Developmental; otherwise, they are classified as Non-Developmental.
Traditional & Nontraditional-Age
Item 30: “Mark your age group.”
Respondents under age 18 are excluded from all data sets. Respondents marking age groups 18 to 19, 20 to 21, and 22 to 24 are classified as Traditional-Age and those marking age groups 25 to 29, 30 to 39, 40 to 49, 50 to 64, or 65+ are classified as Nontraditional-Age.
First-Generation & Not First-Generation
Item 38: “Who in your family has attended at least some college?”
If the respondent indicated that either their mother or father had attended at least some college, then the student is classified as Not First-Generation; otherwise, they are classified as First-Generation.
Male & Female (Sex)
Item 29: “Your Sex.”
Race/Ethnicity
Item 35: “What is your racial/ethnic identification?”
Not Online-Only & Online-Only Students
Online Survey Item: During the current academic term, how many classes are you taking...(a) Face-to-face, (b) Online, (c) Hybrid
If the student took the survey on-paper (in a classroom) or if they indicate taking at least one face-to-face or hybrid class, they are categorized as “Not Online-Only.”
Survey Type (available only to colleges that administered both on-paper and online)
If a student took the survey on paper in a randomly selected classroom, they are classified as In-Class Survey. If a student took the online version of the survey, they are classified as Online Survey.
Student Identifier Data
In accordance with Texas state law and The University of Texas at Austin's policies, CCCSE does not provide student identifiers in the raw data file available for download via the SENSE online reporting system. To request a securely transmitted data file with student identifiers, please contact your SENSE liaison or surveyops@cccse.org.
Back to top