Click here to download this policy brief as a PDF.

New York’s prosperity, today and in the years to come, will largely depend on the quality of its workforce. That’s why the state spends billions of dollars each year on educating and training future workers through the PK-12 system, public colleges and universities, workforce development programs and adult literacy programs. However, very little is known about how effective these programs are in moving participants into careers with life-sustaining wages. This is not because of a lack of information, but because state law has traditionally prevented most local agencies and education providers from using wage data from the Federal Unemployment Insurance (UI) program to track the progress of graduates.

A new law going into effect in 2014 gives local workforce agencies, community colleges, school districts and others the right to access this data for the first time. This is a huge step forward for the state’s human capital system. Among other things, it will allow individuals and policymakers alike to understand which educational and workforce development programs are producing the best outcomes for participants. With this information, New Yorkers would be able to enroll in the workforce development programs that have a proven record of success, while state and local officials could better align limited public resources with the most successful programs.

But to make the most of this opportunity and bring New York State practices in line with what other states are doing, Governor Cuomo and the New York State Legislature should go beyond merely providing access to UI data and give local agencies and providers the expertise and administrative capacity they need to effectively use this data. 

The vast majority of agencies and providers across the state, particularly outside of New York City, have never used UI data before, or created and interpreted the longitudinal studies that incorporate it. And according to experts we interviewed for this policy brief, creating effective program assessments requires significant IT capacity and scientific expertise, in addition to any up-front costs of redesigning programs based on their results.

New York would do well to follow the examples of several states, including Florida, Maryland and Washington, that have overcome these obstacles by creating a statewide office and data warehouse that could help local agencies and providers conduct and interpret their own assessments. This would allow all agencies regardless of size equal access to program evaluation while allaying privacy concerns around individual wage information. It may also open the door to smarter state-level economic development strategies that build coherently off of regional human capital assets.

“Until now, workforce development providers and funders have not had access to wage reporting system data that can provide a comprehensive way to evaluate which programs work and which don’t. Solving this piece of the puzzle was critical. Without this data, judgments could not be made on the strength of public, private and nonprofit training programs across the city.”

Last year, human capital development programs in New York State spent $450 million from federal, state, local, and private sources; community college programs through CUNY and SUNY spent an additional $3 billion. The goal of all these programs is to help people become more employable by equipping them with the skills and social supports they need to get and retain employment, increase their earnings, and become self-sufficient. But in the absence of any data on how the participants of these programs are doing in the labor market, it is impossible for educators and workforce development practitioners to know for sure what kind of an impact they’re having.

“Until now, workforce development providers and funders have not had access to wage reporting system data that can provide a comprehensive way to evaluate which programs work and which don’t,” said Merrill Pond, a senior vice president at the Partnership for New York City. “Solving this piece of the puzzle was critical. Without this data, judgments could not be made on the strength of public, private and nonprofit training programs across the city.”

“We are going to be able to look at earnings of our graduates, for the first time ever,” adds Suri Duitch, university dean for continuing education at CUNY. “That’s really important.”

To establish effectiveness, educators need to know not only whether students got a job after graduation but whether they stayed in a job and were able to move into higher paying positions. If a program is effective, then its participants should do better on these metrics than similarly situated residents who never went through the program.  

As it happens, the New York State Department of Labor (DOL) already has all the information evaluators need to make these determinations. Unemployment insurance (UI) wage records exist for almost every employee in the state, and because they are indexed by Social Security number they can be linked to an individual’s K-12 school records, college transcripts, record of participation in human capital development programs, and any other piece of information about the individual that is indexed by Social Security number. All of this can provide a detailed picture of how participants in specific educational programs are faring in the labor market.

“What this gives you, even if it is retrospective, is data on your human resources supply chain,” says Davis Jenkins, a senior research associate at the Community College Research Center at Columbia University. “It’s astounding to me that states haven’t used that information more.”

As an example, the administrators of a training program for physician’s assistants could use UI data to see whether their program helped increase alumni earnings, and whether it helped them get employment more quickly and keep it for a longer period of time compared to a control group. Similarly, people who are shopping around for a physician assistant training program would be able to look at aggregate program performance information and see how alumni of each program fared in the job market. Having access to longitudinal data on workforce outcomes would even enable local governments to formulate sector-based economic development strategies.

“If an employer went up to Governor Cuomo and said, ‘I can hire 500 welders here in this upstate county over the next three years,’ Cuomo couldn’t tell them how many people are trained as welders in New York State, how many are out of work and are looking for welding jobs, how many people went through welding training programs, and how many are grads of programs that are successful in training welders,” says Andy Van Kleunen, executive director of the National Skills Coalition, a national workforce development advocacy organization..

Before the recent legislation, the use of UI data for human capital assessments was rare in New York. Although local Workforce Investment Boards (WIBs) had the ability to apply to the state Department of Labor to gain special access to the records, not a single WIB had a memorandum of understanding with the DOL, and only a select number of agencies statewide ever used the data in their program assessments. The state’s community colleges and other non WIA-funded human capital stakeholders, meanwhile, were all but barred from using this important data resource and had to resort instead to expensive and ineffective alumni surveys.

Assembly Bill 7911—which goes into effect on December 21st—finally changes all of that. For the first time, local Workforce Investment Boards, Community Colleges, four year colleges, school districts and others will have a much easier time using UI data in their program effectiveness assessments. At the same time, however, if these sorts of assessments are going to become a routine part of the state’s workforce development efforts, giving local agencies access to the data won’t always be enough by itself, particularly outside of New York City where they tend to have fewer employees and financial resources.

In order to have a truly accountable human capital system, many agencies and educational providers across the state will need financial and administrative support to create evaluations and interpret their results, redesign programs and create long-term economic development strategies that better align workforce assets with opportunities in the economy. Longitudinal studies can be complex and expensive to implement. Many of the agencies that have already started to design and use them tend to hire evaluation consultants to provide the brainpower, though they still need to dedicate staff resources to work with the consultants.

Moreover, making individual agencies and organizations across the state design and implement their own studies without any direction or input from state officials is liable to be expensive and wasteful.

“The wrong thing to do would be to have individual groups doing this all by themselves,” says Merrill Pond of the Partnership for New York City. “Individual workforce development groups should be focused on service delivery, not on pulling WRS data and doing research.”

One agency director who has experience using this data told us that it took four to six months just to build up the administrative capacity to interpret the results and implement changes. “You need the kind of data scientists who are appropriate to be working with this data,” she says, “as well as one-time capital to create the IT infrastructure.”

Lessons from Florida, Maryland and Washington

Twenty-eight different states nationwide have already started to use UI data to measure the effectiveness of their workforce programs. And several of these states have gone beyond providing access to UI data to create centralized offices with expertise in wage tracking. In different ways, Florida, Maryland and Washington have all invested in the extra administrative capacity that local agencies and educational providers need in order to implement and interpret the kind of longitudinal studies that could make a big difference in their human capital investments.

The Washington State Workforce Training and Education Coordinating Board, for instance, conducts effectiveness studies for all of its workforce programs every two years. Longitudinal data systems link educational records with UI data to provide a detailed look at how former participants are faring in the labor market.  For each program or provider, the studies show how many of the participants got and kept a job for up to two years after finishing, how much they earned on average, what percent earned a credential, and their satisfaction level with the program. To show the difference the program makes for individuals, researchers in Washington compare the employment and earnings outcomes of participants to the outcomes of control groups comprised of people who did not participate. Finally, the Board calculates a return on investment both for participants and for taxpayers. A summary of each evaluation is publicly available on the Workforce Board’s website, allowing taxpayers, policymakers, and potential participants to see which programs are most effective.

Washington State has been able to use their evaluation results to improve program offerings and restructure inefficient programs. For example, when research revealed that less than half of the participants in adult basic education programs obtained employment and that there was no net positive benefit in return on investment to either participants or taxpayers, the state began to look at why the programs were failing to meet their goals. After convening regional experts in economic development, workforce, commerce, education, and higher education, the Washington State Board for Community and Technical Colleges (SBCTC) created and tested a pilot program which eventually became the Integrated Basic Education and Skills Training (I-BEST) program. This program has now been implemented nationwide and has dramatically improved outcomes for adult basic education participants in the state and across the nation.

The SBCTC has also been able to use its UI data research to establish a funding system for community colleges called the Student Achievement Initiative that rewards individual colleges for helping students achieve milestones towards completion of their degree programs or certificates. The Board’s research documented a “tipping point” in educational achievement; people who completed at least a one year certificate program, regardless of where they began their educational pathway, showed a considerable advantage in earnings over non-completers from their same cohorts. Dr. David Prince, the policy research director at SBCTC and one of the authors of the study, told us that program completion was the single most important factor for student success. “Over and over again,” he says “we see that completion in and of itself is important. We can see it in the data.”

Whether they are associate degree programs at state community colleges or six-month certificate courses, reducing the number of students who drop out or otherwise fail to graduate within in a reasonable amount of time is one of the surest ways to increase the state’s return on investment. This insight was only made possible by creating studies that link UI data with educational records.

In Maryland, the Jacob France Institute (JFI) at the University of Baltimore is authorized by the state to serve a similar role to Washington’s State Training and Education Board. The Institute serves as a repository for wage and labor market data that can be used in longitudinal studies and works with local agencies and organizations across Maryland to design and implement these studies. JFI even has data sharing agreements with several nearby states in order to be able to track students and workers who cross state borders.

JFI worked with the Baltimore public school system to follow a cohort of high school dropouts from the mid-1990s through seven years in the labor market. The results showed dramatic earnings level differences between dropouts and graduates. The Baltimore public schools incorporated this information into their guidance counselor materials, showing students at risk of dropping out what the consequences would be. JFI also worked with several apprenticeship programs in the state and was able to quantify the earnings advantage that accrues to those who complete the program versus those who do not. The Institute also does similar research using UI data to calculate how much of an impact community college programs have on the earnings of their graduates.

JFI established its role as the clearinghouse for longitudinal education and workforce data through the efforts of its executive director David Stevens, who along with a handful of other researchers working around the country in the 1970s was one of the first to explore the value of UI data for doing evaluation of workforce and education programs. According to Treva Stack, an analyst who works with UI data at JFI, the work that the State of Maryland is able to do with this information is due to the trust that Mr. Stevens was able to garner over the years through being a responsible steward of the data. “The biggest challenge is trust,” said Ms. Stack. “People don’t like to lose control over their data. Because we have been doing this for so long, they trust us. We don’t release results to anyone but the clients. Clients are in control of the data and their findings.”

The Florida Education and Training Placement Information Program (FETPIP) is a data collection center housed in that state’s Department of Education that collects data from universities, community colleges, schools, and other educational and workforce development programs, as well as demographic information and wage records, to create longitudinal datasets indexed to specific individuals. The matched records are kept on protected computers at FETPIP, where only a few staff have access to them.

FETPIP was authorized in 1988 through state legislation to “compile, maintain, and disseminate information concerning the educational histories, placement and employment, enlistments in the United States armed services, and other measures of success of former participants in state educational and workforce development programs.”  Duane Whitfield, the founding director of FETPIP reported that employers were getting inundated with requests from school boards, community colleges, and universities for follow-up information on their students, so when the state switched to using the administrative wage records in place of going to employers themselves it was a relief. It also proved to be cost-effective for the state. “The use of administrative wage records makes follow-ups so cheap and so much more comprehensive than any kind of surveys,” says Mr. Whitfield. “Surveys were costing our research lab $17 each to process, but when we switched to using wage records it cost about a penny or two per record. It’s a humongous savings.”  Furthermore, Mr. Whitfield reported that Florida made the decision to house FETPIP at the state department of education because it would get around FERPA restrictions on sharing student information with outside entities.

All three of these nationally-recognized agencies have established reputations for protecting confidential data; indeed, the fact that the wage matching work is centralized means that records linked to individuals pass through fewer hands and can therefore be more easily protected from unauthorized access. While using individually identifiable wage records for research presents important privacy concerns, establishing a central repository and research office, as these three states have done, goes a long way toward allaying some of those concerns.

These models differ in other ways. Washington and Florida have central repositories that are authorized by the legislature to receive data from the agencies that own it, while Maryland depends on individual MOUs with data owners to acquire the data. Authorizing central repositories to collect, store, and do research with unemployment insurance wage data by statute protects the system to a degree against being dismantled if a new political administration with less interest in the work takes office. 

Maryland offers a model of a system that uses cross-state data sharing agreements that allow each state to track workers who move across state lines during the course of their educational and workforce pathways. However, access to the data across state borders is highly dependent on personal relationships with owner agencies in each of the partner states. In addition, while Florida and Washington State make the results of its evaluations of human capital development programs public, Maryland releases the results only to their client agencies.

Limitations of Wage Tracking

Longitudinal studies using UI wage data cannot answer all the questions that human capital development programs need to answer to determine how they may improve their programs. For some areas of study, particularly at the bachelor’s degree level and higher, relative wage levels may not be very useful for either students or educators, though the vast majority of workforce development programs and professional programs at community colleges are serving students who want to enter a career with living wages, making earnings a useful standard of success.  

More problematically, UI data does not contain information on occupations, so the data may show that a graduate of a radiology training program is working for an employer that is classified under health care but not whether they are working as a radiologist’s assistant or as an administrative assistant in a radiologist’s office.

Also, the data reports wage levels by quarter in the aggregate, so there is no way to distinguish how much of an individual employee’s wages came from base salary, overtime benefits, bonuses, or other sources. This lack of specificity makes it difficult to compare wage levels across employees and employers. Finally, wage data from self-employed individuals and agricultural, federal, railroad, and military workers are not reported through the unemployment insurance system, so alumni and former participants of workforce development programs employed in these areas of the labor market cannot be tracked. Federal workers counted under a different data system called FEDES, to which some states, including Maryland, have negotiated access.

David Pavelcheck of the Washington State Workforce Training and Education Coordinating Board cautions that when translating the evaluation results into action, policymakers should be careful to recognize that the data only provide a simple picture about educational and workforce pathways, and that at best they can point to areas in the human capital development system that need improvement. “Everyone wants the data to point out simple and clear instructions,” he cautions. “Data can’t be less complicated than reality, and reality is very complicated. People come in [to my office] saying ‘tell me which programs are less efficient than others and I will kill them in a New York minute.’  These programs serve different client groups for different purposes, so cutting the ones that are deemed less efficient is not always what you want to do. But people don’t want to hear that.”

Nonetheless, most of the researchers and program administrators to whom we spoke considered UI wage records to be the gold standard source for tracking individual workforce outcomes, especially because there are almost no reliable alternative sources. Specifically, UI wage record data has the advantage of being available nationwide and being indexed by Social Security number, which provides an ideal individual identifier to make it more straightforward to match the data with that other datasets that are also indexed with Social Security numbers. It is also relatively inexpensive compared to sending out surveys or contacting employers individually to get the information.

Giving agencies the capacity to use UI wage data for research is the key to ensuring that Governor Cuomo’s first step in opening access to data becomes a giant leap forward for the ability of the state’s human capital development programs to evaluate program effectiveness and develop long-term economic strategies that build off of local workforce assets.

New York City has taken the lead through the Office of Human Capital Development (OHCD) in expanding access to UI wage records, and is also the part of the state that is most likely to benefit from access to these data. While New York City agencies such as OHCD, the Center for Economic Opportunity and CUNY already have the capacity to use this information for program evaluation, agencies in other parts of the state are less likely to be as well equipped to do so.

Therefore, the very best way the state can ensure that all human capital programs in the state have equal access to evaluation services is to follow Florida’s model and establish a central state-level office funded with general revenues. Like Maryland, it should establish data sharing agreements with nearby states to capture the outcomes of New Yorkers who seek employment outside the state. Finally, New York should follow Washington State’s model of conducting regular evaluations and making summary results available to the public.

After all, opening access to evaluation data is only as powerful as the state’s human capital development programs’ ability to leverage the data to improve its programs.