Source Notes
Each year, the National Science Foundation (NSF) collects data from hundreds of academic institutions on expenditures for research and development in science and engineering fields and classifies them by source of funds (e.g., federal government, state and local government, industry, etc.). These data are the primary source of information on academic research and development (R&D) expenditures in the United States. Included in this survey are all activities specifically organized to produce research outcomes that are separately budgeted and accounted for. This “organized research” may be funded by an external agency or organization (“sponsored research”) or by a separately budgeted organizational unit within the institution (“university research”). This report excludes activities sponsored by external agencies that involve instruction, training (except training in research techniques, which is considered organized research), and health service, community service or extension service projects.
All Federally Funded Research Labs (FFRLs) are excluded from these academic expenditures data, including the following: Jet Propulsion Laboratory (California Institute of Technology); Los Alamos National Lab, Lawrence Livermore Lab, Lawrence Berkeley Lab (University of California); Software Engineering Institute (Carnegie Mellon); Argonne National Laboratory (University of Chicago); National Astronomy and Ionospher Center (Cornell); Ames Laboratory (Iowa State University); Lincoln Laboratory (MIT); Plasma Physics Lab (Princeton); and SLAC National Accelerator Laboratory (Stanford). The NSF data no longer classify the Applied Physics Lab (APL) at Johns Hopkins as an FFRL, but federal funds support the vast majority of research conducted there. The APL makes up more than one-half of Johns Hopkins’ total and federal R&D expenditures.
While inconsistencies in reporting (known and unknown) do exist here, as in any survey of this type, problems arise mostly when one breaks out the data by source of funds. NSF expects institutions to use year-end accounting records to complete this report, and there are nationally recognized accounting guidelines for higher education institutions. However, there are also countless variations in institutional policy that determine whether the university reports a particular expenditure as coming from one source or another, or possibly not counted at all. Take federal formula funds for agriculture (e.g., Hatch-McIntire, Smith-Lever) as an example. We conducted an informal survey of the appropriate institutions in the Association of American Universities (AAU) and found that two out of eleven land grants did not include any of these federal funds in their 1997 NSF data, while others included all or some of these monies. Because these funds make up a very small percentage of the total research expenditures in any given year, the impact on our total research rankings is slight. The agriculture formula funds will have a somewhat greater, but still small, impact on the federal research rankings. NSF notes, “An increasing number of institutions have linkages with industry and foundations via subcontracts, thus complicating the identification of funding source. In addition, institutional policy may determine whether unrestricted state support is reported as state or as institutional funds.” 1
We believe that the reporting inconsistencies in the data are relatively minor when using the total research expenditures and the federal research expenditures component. Federal and state government audits of institutional accounting make deceptive practices highly unlikely, even though these entities do not audit the NSF data directly. NSF goes to great lengths to verify the accuracy of the data, especially federal expenditure data—checking them against several other federal agencies that collect the same or similar information. In fact, all major federal agencies and their subdivisions submit data to NSF identifying research obligations to universities each year. Historically, the NSF data have tracked very closely the data reported by universities.2 Further, for their National Patterns of R&D Resources series, NSF prefers to use the figures reported by the performers of the work (that is, academic institutions, industry, nonprofits) because they believe that the performers are in the best position to accurately report these expenditures.
In some sections of this report, these expenditure data are deflated to constant 1983 dollars to show real change over time. While NSF uses the Gross Domestic Price (GDP) implicit price deflator in its reports on federal trends in research, we use the Higher Education Price Index (HEPI) because of its narrower focus. Originally developed by Research Associates of Washington and currently managed by Commonfund Institute, the HEPI illustrates the effect of inflation on college and university operations.3 In contrast, the GDP implicit price deflator is based on change in the entire U.S. economy and, as noted by NSF itself, “[its] use more accurately reflects an ‘opportunity cost’ criterion rather than measure of cost chanages in doing research. That is, the GDP deflator, when applied to R&D expenditure or funding data, reflects [the value of R&D in terms of the amount of other goods and services that could have been purchased with the same amount of money].,”4
The research trend data always reflect the most recent published data available because NSF allows institutions to submit revised figures for up to two years. Each year, NSF reports data for the current year as well as for the previous seven years. NSF’s published nationwide totals for academic R&D expenditures will not always match the corresponding totals in this study due to NSF's sampling procedures for smaller or non-reporting institutions. In some years, rather than identifying the institutions individually, NSF provides one aggregate figure for all sampled institutions.
__________________________
1. Academic R&D Expenditures, FY 2009: Technical Notes (On-line: http://www.nsf.gov/statistics/nsf11313/content.cfm?pub_id=4065&id=3 )
2. National Patterns of R&D Resources, 1998: Technical Notes (On-line: http://www.nsf.gov/sbe/srs/nsf99335/appa.htm )
3. About HEPI, Commonfund Institute (On-line: http://www.commonfund.org/CommonfundInstitute/HEPI)
4. National Patterns of Research Development Resources, 2003: Technical Notes (On-line: http://www.nsf.gov/statistics/nsf05308/appa.htm )
Endowment Assets
Source: NACUBO Endowment Study as reported in the Chronicle
of Higher Education
Institutions report the market value of their endowment assets as of June 30 to three different sources, and they quite often use three different values. For this project, we use the National Association of College and University Business Officers (NACUBO) Endowment Study because of NACUBO’s long history of reporting endowments of higher education institutions, their emphasis on using audited financial statements, and their focus on net assets (i.e., includes returns on investments and excludes investment fees and other withdrawals). NACUBO conducts its study annually and reports the results each February in the Chronicle of Higher Education.
Another source for endowment assets is the Council for Aid to Education’s (CAE) annual Voluntary Support of Education (VSE) survey, cosponsored by the Council for Advancement and Support of Education (CASE) and the National Association of Independent Schools. The VSE survey is useful as a secondary resource because it provides more single-campus data than the other two sources. For those institutions that report a system-wide total to NACUBO, we often use the VSE data to calculate a campus’ percentage contribution to the entire system, applying that factor to the NACUBO figure. In other cases, we may substitute the VSE figure when the institution indicates that this is a good data source.
The National Center for Education Statistics (NCES) Integrated Postsecondary Education Data System (IPEDS) Finance Survey also collects information on endowment assets; but these data are available much later. In our inaugural report of The Top American Research Universities in 2000, we noted the wide variation in the reporting of endowment market value between all three sources. An examination of the 1997 endowment figures showed only one university (University of North Carolina at Chapel Hill) had submitted the same figure to each of the three organizations. In a more recent study of major research universities, we found about one-third of the all institutions report identical figures but just seven universities in our over $40 million federal research group. In the earlier study, we found that endowment assets reported to IPEDS tended to be lower than NACUBO or VSE data, but this is no longer true. In general, the greater the endowment the likelihood that the figures reported to the three sources will vary. Both studies found no consistent pattern to explain reporting variations among the institutions.
Annual Giving
Source: Council for Aid to Education’s Voluntary
Support of Education (VSE) Survey
The Council for Aid to Education (CAE), an independent subsidiary of RAND, has produced the Voluntary Support of Education (VSE) Survey since 1986. The annual giving data include all contributions actually received during the institution’s fiscal year in the form of cash, securities, company products, and other property from alumni, non-alumni individuals, corporations, foundations, religious organizations, and other groups. Not included in the totals are public funds, earnings on investments held by the institution, and unfulfilled pledges.
CAE's VSE Data Miner service, available online, provides 10 years of data for all participating institutions (more than 1,600 universities and colleges). Although this is a subscription-based service and requires a user ID and password, limited access is available online at
[http://www.cae.org/vse].
National Academy Members
Source: National Academy of Sciences, National Academy of
Engineering, and Institute of Medicine membership online directories
One of the highest honors that academic faculty can receive is membership in the National Academy of Sciences (NAS), the National Academy of Engineering (NAE), or the Institute of Medicine (IOM). All three are private, nonprofit organizations and serve as advisors to the federal government on science, technology, and medicine. Nominated and voted on by active members, newly elected members of these organizations receive life terms. Individuals elected to membership come from all sectors—academia, industry, government, and not-for-profit agencies or organizations. Member election dates are in February (NAE), April (NAS), and October (IOM).
The data collected for these rankings use active or emeritus members at their affiliated work institution, as reported in the online membership directories. In all cases, we were able to determine the specific campus for individual members. We re-check institutional affiliation annually to account for established members who have changed employers or whose membership is no longer active.
Faculty Awards in the Arts,
Humanities, Science, Engineering, and Health
Source: Directories or web-based listings for multiple agencies
or organizations.
For this category, we collect data from several prominent grant and fellowship
programs in the arts, humanities, science, engineering, and health fields.
Included in this measure are:
• American Council of Learned Societies (ACLS) Fellows
• Beckman Young Investigators
• Burroughs Wellcome Fund Career Awards
• Cottrell Scholars
• Fulbright American Scholars
• Getty Scholars in Residence
• Guggenheim Fellows
• Howard Hughes Medical Institute Investigators
• Lasker Medical Research Awards
• MacArthur Foundation Fellows
• Andrew W. Mellon Foundation Distinguished Achievement Awards
• National Endowment for the Humanities (NEH) Fellows
• National Humanities Center Fellows
• National Institutes of Health (NIH) MERIT (R37)
• National Medal of Science and National Medal of Technology
• NSF CAREER awards (excluding those who are also PECASE winners)
• Newberry Library Long-term Fellows
• Pew Scholars in Biomedicine
• Presidential Early Career Awards for Scientists and Engineers (PECASE)
• Robert Wood Johnson Policy Fellows
• Searle Scholars
• Sloan Research Fellows
• US Secretary of Agriculture Honor Awards
• Woodrow Wilson Fellows
While the vast majority of these programs
clearly identify a particular campus, in a few instances we used the institution’s
web-based phone directory to determine the correct campus.
Doctorates Awarded
Source: NCES IPEDS Completions Survey, doctoral degrees
awarded between July 1 and June 30.
Each year, universities report their degrees awarded to the NCES
in the IPEDS Completions Survey. IPEDS provides straightforward
instructions for reporting doctoral degrees awarded, and we do
not find any inconsistencies in reporting among the universities
included in our rankings. IPEDS asks each institution to identify
the number of Doctor of Education, Doctor of Juridical Science,
Doctor of Public Health, and Doctor of Philosophy degrees awarded
between July 1 and June 30.
Most institutions in our study submit degree data by campus or
offer doctoral degrees solely or primarily at the main campus.
In addition to doctorate degrees, The MUP Center presents degrees awarded at other levels—associate’s, bachelor’s, master’s, and professional degrees—in the Student Characteristics table.
Postdoctoral
Appointees
Source:NSF/Division of Science Resource Statistics (SRS) Survey
of Graduate Students and Postdoctorates in Science and Engineering.
Each year, NSF and NIH collect data from all institutions offering graduate programs in any science, engineering, or health field. The Survey of Graduate Students and Postdoctorates in Science and Engineering (also called the Graduate Student Survey or GSS) reflects graduate enrollment and postdoctoral employment at the beginning of the academic year. Postdoctorates are defined in the GSS as “individuals with science and engineering PhD’s, MD’s, DDS’s or DVM’s and foreign degrees equivalent to U.S. doctorates who devote their primary effort to their own research training through research activities or study in the department under temporary appointments carrying no academic rank.” The definition excludes clinical fellows and those in medical residency training programs unless the primary purpose of their appointment is for research training under a senior mentor.
In the methodological notes for this survey,5 NSF indicates that it verifies the data with the institutional coordinator when dramatic year-to-year fluctuations are noted. In addition, efforts have been made recently by NSF to reduce the number of institutions that assigned a zero when they should have recorded it as non-response. Some historical data may have a zero when it should be blank (for non-response) even though The MUP Center has been trying to fix these errors for the past several years.
Although each doctorate-granting campus submits data separately, NSF often aggregates them in its published reports. In all cases, we obtain the single-campus data for these schools directly from NSF.
__________________________
5. Survey Methodology: Survey of Graduate Students and Postdoctorates
in Science and Engineering (On-line: http://www.nsf.gov/statistics/srvygradpostdoc/)
SAT Scores
Source: NCES IPEDS Survey, SAT and ACT Scores.
IPEDS reports the 25th and 75th percentiles for verbal and
quantitative SAT I scores for most institutions in our study. For our measure, we calculated the median of that
range. Some institutions report the ACT instead of the SAT to IPEDS and some report both. We select the test which has the greatest percentage of students reporting. To convert ACT scores, we use a conversion table provided by The College
Board to generate a comparable SAT equivalent score.6 When an institution
submits neither an SAT nor ACT score, we substitute data from other national data sources.
__________________________
6.ACT and SAT Concordance Tables, November 6, 2009 (On-line:
http://www.collegeboard.org/research/html/rn_indx.html)
Other Measures of Undergraduate Quality
National Merit and Achievement
Scholars
Source: The National Merit Scholarship Corporation
Annual Report
The National Merit Scholarship Corporation (NMSC) is an independent, nonprofit organization that awards scholarships to the nation’s outstanding high school seniors based on their academic achievement, qualifying test scores, high school principal and counselor recommendations, and their activities, interests, and goals. The NMSC names approximately 15,000 National Merit Finalists each February. Of these, about 8,000 will receive a National Merit $2,500 Scholarship, a corporate-sponsored scholarship, or a college-sponsored scholarship.
National Achievement Scholars are selected and funded in a similar fashion and represent the nation’s outstanding African-American students. Ideally, the National Hispanic Scholars Program should also be included in this category, but it does not track the enrollment of its scholarship winners. Should it do so in the future, we will include these students in The MUP Center's data. In this study, Merit and Achievement scholarships are credited to the main campus if the NMSC Annual Report does not indicate a branch campus.
While the number of National Merit and National Achievement award winners in the entering class provides an indication of the attractiveness of a university’s undergraduate program to outstanding students, it is also an indicator that is sensitive to institutional policies on financial aid. Because the number of Merit Scholars is small, relatively small changes in institutional aid policies can have a significant impact on the number of National Merit Scholars enrolling in institutions. The average SAT score provides a broader-based and more reliable measure of overall undergraduate quality; for those reasons, we prefer the SAT scores to the number of National Merit and Achievement Scholars as an indicator of undergraduate quality.
Medical Schools
Source: NCES IPEDS Completions Survey, MD degrees awarded.
Although the IPEDS Institutional Characteristics Survey does have a “medical” field that indicates whether an institution grants a medical degree, we choose not to use its data because it includes medical degrees in Veterinary Medicine, Dentistry, and other professional health-related fields. For our measure, we determined whether a particular campus awarded any MD degrees during the academic year. If the institution did not submit any data to IPEDS for that year, we then looked at whether it was accredited by the American Medical Association to determine whether the institution has a medical school.
Land Grant Institutions
Source: National Association of State Universities and Land
Grant Colleges.
The first Morrill Act in 1862 appropriated federal funds for universities to provide agricultural and technical education to their citizens. A second Morrill Act in 1890 expanded eligibility to include several historically black colleges and universities, and in 1994 several Native American tribal colleges were recognized as land grant institutions. Today, there is at least one land grant institution in each state and U.S. territory and in the District of Columbia. Of the 105 institutions, most are public universities. Federal land grant institutions receive both federal and state dollars in support of their agricultural and extension activities.
While land grant status technically applies to some university systems, such as the University of California and the University of Nebraska, for our study we designate as land grant institutions only those schools that actually perform that function (e.g., UC-Berkeley, UC-Davis, UC-Riverside, Nebraska-Lincoln). In these cases, the land grant field will identify whether an institution is part of a system-wide land grant and whether the vast majority of the activity occurs on that campus. For example, UC-Davis is coded as “Yes-System” while UCLA is coded as “No-System.” We consider the 1890 institutions as land grant institutions, but we identify them separately because they do not perform extension activities.
Research Focus
NSF/SRS Survey of R&D Expenditures at Universities and Colleges.
In addition to reporting expenditure data by source of funds, NSF identifies in what major disciplines the money is expended. In the Research by Discipline table we provide the p4oportion of total and federal expenditures in each discipline for those institutions with more than $20 million in federal research. These data are useful for developing groups of similar institutions for peer analysis.
The Institutional Characteristics table provides a summary measure of an institution’s research strength and concentration based on these discipline-level expenditures. Universities with 95-100% of their federal research dollars spent in one particular discipline are coded as “all.” We identify institutions with 75-94% in one area as “heavy” and label those with 50-74% of their expenditures concentrated as “strong.” Other universities with 25-49% in one or more disciplines we describe as “moderate.” A few institutions (but none in the more than $40 million group) have expenditures distributed fairly evenly across the disciplines; those we code as “mixed.”
In some cases, where an institution reports as a multi-campus entity, we made adjustments to break out the discipline-level expenditure data by single campus. Typically, this involved moving all or a portion of the life sciences expenditures to the health or medical center campus. IPEDS fall enrollment and graduate degrees by discipline data also were used to help in this effort.
While these data offer some insight as to the research structure of a university, their usefulness is limited. For example, we may be tempted to use the life sciences as a surrogate for medical research, but we must remember that they also include agricultural and biological sciences. Further, the growing trend toward multidisciplinary and interdisciplinary projects may make it more difficult for universities to accurately reflect expenditures by discipline or sub-discipline.The MUP Center chooses not to break out these sub-disciplines because the data are increasingly prone to error as further adjustments are made.
Fall
Enrollment
Source: NCES IPEDS Fall Enrollment Survey.
Each November, institutions report their current fall headcount enrollment to the IPEDS Fall Enrollment Survey. Enrollment figures include both degree seeking and non-degree seeking students.The MUP Center provides the headcount enrollment by level as presented by IPEDS, along with the percentage of those attending part-time. Graduate students include those seeking specialist degrees in engineering and education. First professional students include those seeking degrees in medical fields, such as Chiropractic, Dentistry, Medicine, Optometry, Osteopathic Medicine, Pharmacy, Podiatry, and Veterinary Medicine, as well as those seeking degrees in Law and Theology.
Each campus in our study submits enrollment data by campus, except for the few institutions identified in each year's Top American Research University publication. Because this is an informational item and not one of The MUP Center's nine quality measures, we did not attempt to adjust these figures.
Federal Research with and without Medical School Research
AAMC Federal Research
Source: Association of American Medical Colleges
The Association of American Medical Colleges collects data on federally sponsored research at medical colleges through on the Liaison Committee on Medical Education (LCME) Part I-A, Annual Financial Questionnaire. We calculate each medical school’s federal R&D by summing the recorded dollars and a portion of the relative administrative costs. We exclude the not-recorded dollars.