TSA Should Issue More Guidance to Airports and Monitor Private versus Federal Screener Performance

Dec. 7, 2012
Questions about the cost-effectiveness and screening efficiency of private screeners compared with that of federal screeners have increased scrutiny of the SPP

GAO-13-208

Screening Partnership Program: TSA Should Issue More Guidance to Airports and Monitor Private versus Federal Screener Performance

December 6, 2012

The Honorable John Mica Chairman Committee on Transportation and Infrastructure House of Representatives

Dear Chairman Mica

The Transportation Security Administration (TSA) is responsible for ensuring the security of the traveling public through, among other things, screening passengers traveling by aircraft for explosives and other prohibited items. To fulfill this responsibility, TSA maintains a federal workforce of screeners at a majority of the nation`s commercial airports, but also oversees a smaller workforce of private screeners employed by companies under contract to TSA at airports that participate in TSA`s Screening Partnership Program (hereafter referred to as the SPP, or the Program).[Footnote 1] The SPP, established in 2004 in accordance with provisions of the Aviation Transportation Security Act (ATSA), allows commercial airports an opportunity to ``opt out`` of federal screening by applying to TSA to have private-sector screeners perform the screening function.[Footnote 2] At private screening airports, TSA continues to be responsible for overseeing airport screening operations and ensuring that the contractors provide effective and efficient security operations in a manner consistent with law and other TSA requirements; however, the screening of passengers and baggage at these airports is performed by private screening contractors selected and approved by the TSA.[Footnote 3]

Questions about the cost-effectiveness and screening efficiency of private screeners compared with that of federal screeners have increased scrutiny of the SPP. Some representatives of the aviation industry and certain airport operators contend that the SPP capitalizes on the private sector`s innovation and flexibility to provide screening services more efficiently and with enhanced customer service. However, the SPP model allows for little variance in screening operations at SPP airports. For example, in accordance with ATSA, private screeners must meet the same standards and requirements for hiring and training that apply to federal screeners, abide by the same standard operating procedures, and be provided compensation and benefits at a level not less than their federal counterparts.[Footnote 4]

As of January 2011, 16 airports were participating in the SPP. [Footnote 5] In January 2011, the TSA Administrator announced his decision not to expand the SPP beyond the 16 participating airports ``unless a clear and substantial advantage to do so emerges in the future.`` In so doing, the Administrator cited his interest in helping the agency evolve into a ``more agile, high- performing organization that can meet the security threats of today and the future`` as the reason for his decision. Of the 6 airports that submitted applications from March 2009 through January 2012 that were evaluated under the ``clear and substantial advantage`` standard, TSA approved the application of 1 airport, West Yellowstone Airport, and denied the applications of the other 5 airports. According to TSA officials, however, the Federal Aviation Administration (FAA) Modernization and Reform Act of 2012 (FAA Modernization Act), enacted in February 2012, prompted TSA to change the standard by which it evaluates SPP applications and requires, among other things, that the TSA Administrator approve an SPP application submitted by an airport operator if the Administrator determines that the approval would not compromise security or detrimentally affect the cost- efficiency or the effectiveness of the screening of passengers or property at the airport.[Footnote 6]

Citing the recent SPP airport approvals, the Committee on Appropriations, House of Representatives, recommended approximately $160 million for privatized screening for fiscal year 2013, which represents $15 million above the amount requested in the Department of Homeland Security`s (DHS) budget request and about $14 million above the enacted fiscal year 2012 level.[Footnote 7] The report noted that the funding was increased to ensure adequate resources to support potential new SPP participants and to encourage TSA to make greater use of the program. The report of the Committee on Appropriations of the Senate, however, recommended approximately $143 million for privatized screening for fiscal year 2013, the same amount requested in the budget and approximately $1 million below the enacted fiscal year 2012 level.[Footnote 8] In its report, the Senate Committee explained that it expects TSA to not approve any new contract application for privatized screening if the annual cost of the contract exceeds the annual cost to TSA of providing federal screening services at that airport.

We reported in January 2009, among other things, that TSA had underestimated costs to the government for screeners at non-SPP airports because the agency did not include all of the costs associated with passenger and baggage screening services at these airports.[Footnote 9] The omission of some cost factors reduced the reliability of TSA`s 2009 cost estimate by increasing the costs for private-contractor screeners relative to federal screeners. We recommended that if TSA planned to rely on its comparison of cost and performance of SPP and non-SPP airports for future decision making, the agency should update its analysis to address the limitations we identified. TSA generally concurred with our recommendation, and in March 2011, we reported that TSA has made progress in addressing the limitations related to costs and estimated that SPP airports would cost 3 percent more to operate in 2011 than airports using federal screeners.[Footnote 10]

This report addresses the (1) status of airport applications made to the SPP, and airport operator, other stakeholders, and TSA views on the advantages and disadvantages of participating in the SPP; (2) extent to which TSA has provided guidance to govern the SPP application process; and (3) extent to which TSA assesses and monitors the performance of private and federal screeners.

This report is a public version of the prior sensitive report that we provided to you. DHS deemed some of the information in the prior report as Sensitive Security Information (SSI), which must be protected from public disclosure.[Footnote 11] Therefore, this report omits information about the specific results of our comparison of SPP screener performance with performance of federal screeners across four performance measures we analyzed. Although the information provided in this report is more limited in scope, it addresses the same questions as the sensitive report. Also, the overall methodology used for both reports is the same.

To address all three of these objectives, we interviewed Federal Security Directors (FSD); airport operators; screeners; and where applicable, SPP contractors at 10 airports. We selected the 10 airports by matching an SPP to a non-SPP airport, in each of the five airport categories (category X, I, II, III, and IV), based primarily on (1) annual passenger and baggage volumes, (2) screener staffing model full-time equivalent allocation, and (3) number of check-points and screening lanes.[Footnote 12] Additionally, on the basis of available travel resources, we visited 7 of the 10 airports to observe airport screening operations, including any unique challenges faced by these airports. Our observations from these airport visits and interviews are illustrative and provide insights about private and federal screening operations but are not generalizable to all airports across the country.

To determine the status of SPP applications, and airport operators`, other stakeholders`, and TSA`s views on the advantages and disadvantages of participating in the SPP, we interviewed officials of TSA`s SPP Program Management Office (PMO) and reviewed the 15 SPP applications that had been submitted since fiscal year 2009, as well as TSA`s available decision memos on the applications.[Footnote 13] Further, we surveyed the 28 airport operators who have applied to the SPP since its inception through April 2012 (when the survey was implemented) to solicit their views on advantages and disadvantages for airports for participating in the SPP.[Footnote 14] A 29th airport, Bozeman Yellowstone International Airport, applied to the SPP for the first time in June 2012 and, therefore, was not included in our survey. However, we interviewed this airport, as well as the 5 non- SPP airports we visited or interviewed, to obtain their perspectives on the potential advantages and disadvantages of participating in the SPP. We also interviewed representatives of three aviation industry associations to identify the advantages and disadvantages of using federal and nonfederal screeners.[Footnote 15] We selected the three associations because they represent the majority of aviation industry stakeholders, including airport operators.

To determine the extent to which TSA has provided guidance to govern the SPP application process, we analyzed past and current SPP application forms and instructions, as well as interviewed TSA headquarters officials to identify the requirements and process for applying to the SPP. We surveyed operators of all approved SPP airports as well as operators of airports that have applied but are currently not participating in the SPP because TSA denied their application, or their participation is pending the procurement of a contractor, to determine their perspectives on the SPP application process. To determine if any improvements are needed to the SPP application process, we compared TSA`s application process and requirements with Standards for Internal Control in the Federal Government.[Footnote 16]

To determine the extent to which TSA assesses and monitors the performance of private and federal screeners, we interviewed TSA headquarters officials knowledgeable about TSA`s performance management process to identify current screener performance measures. At the airports we visited, we observed screening operations to identify areas where screener performance could be assessed and interviewed contractor, airport, and TSA officials to obtain their perspectives on the current set of performance measures. We evaluated TSA`s process for assessing and monitoring the performance of private and federal screeners against Standards for Internal Control in the Federal Government and best practices for performance management. [Footnote 17] To determine how screener performance compares at SPP and non-SPP airports, we compared screener performance for the 16 currently participating SPP airports to the average performance of other airports in their category, as well as nationally, from fiscal year 2009 through 2011. To ensure the reliability of the performance measure data we analyzed, we (1) interviewed TSA officials who use and maintain the data; (2) checked the data for missing information, outliers, and obvious errors; and (3) reviewed documentation on the relevant data systems to ensure the data`s integrity. On the basis of the steps we took, we found the data reliable for the purpose of providing summary statistics of screener performance for the four performance measures we analyzed. However, as we note later, because there are many factors that may account for differences in screener performance, some of which cannot be controlled for, any difference we found in screener performance at SPP and non-SPP airports cannot be entirely attributed to the use of either federal or private screeners.

We conducted this performance audit from November 2011 to November 2012, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. More details about the scope and methodology of our work are presented in appendix I.

Background

ATSA established TSA and charged it with responsibility for securing all modes of transportation, including civil aviation. Prior to ATSA and the establishment of TSA, passenger and baggage screening had generally been performed by private screening companies under contract to airlines and in accordance with FAA regulations. In accordance with ATSA, TSA currently employs personnel who screen passengers at the vast majority of TSA- regulated (also referred to as commercial) airports nationwide. On November 19, 2002, pursuant to ATSA, TSA began a 2-year pilot program at 5 airports using private screening companies to screen passengers and checked baggage.[Footnote 18] In 2004, at the completion of the pilot program, and in accordance with ATSA, TSA established a permanent program known as the Screening Partnership Program whereby any airport authority, whether involved in the pilot or not, could request a transition from federal screeners to private, contracted screeners. Each of the 5 pilot airports applied and was approved to continue as part of the SPP, and since its establishment, 20 additional airport applications have been accepted by the SPP. Once an airport is approved for SPP participation and a private screening contractor has been selected, the contract screening workforce assumes responsibility for screening passengers and their property and must adhere to the same security regulations, standard operating procedures, and other TSA security requirements followed by federal screeners at commercial airports.

Federal and Private Screening Roles and Responsibilities

TSA`s SPP PMO, located within TSA`s Office of Security Operations (OSO), coordinates with local TSA officials to support an airport`s transition from federal to private screening operations and supports the day-to-day management of the SPP. The PMO facilitates the SPP application process by reviewing SPP applications, organizing SPP application review meetings with other relevant TSA offices, and preparing and routing relevant application documentation to these offices and the TSA Administrator.[Footnote 19] Along with the TSA Office of Acquisition, the office plays a significant role in contract oversight and administration, as well as actively participates in contract source selection processes.

TSA`s FSDs provide day-to-day operational direction for security operations at the airports within their jurisdiction, including those participating in the SPP. However, FSD management responsibilities differ at airports using federal versus private screeners. For example, at airports with a federal workforce, the FSD directly supervises and controls the screening workforce. However, at SPP airports, the FSD has responsibility for overall security but does not have direct control over workforce management; rather the SPP contractor is contractually obligated to effectively and efficiently manage its screening workforce.

The SPP contractor`s responsibilities include recruiting, assessing, and training screening personnel to provide security screening functions in accordance with TSA regulations, policies, and procedures. SPP contractors are also expected to take operational direction from TSA, through the FSDs, to help ensure they meet the terms and conditions of the contract. In addition, SPP contractors are rewarded for identifying and proposing ideas that TSA accepts for possible innovations in recruiting, training, and security procedures, such as the practice of conducting pre-hire orientations to inform prospective screener candidates of the position requirements, which is 1 of over 200 ideas submitted to TSA by SPP contractors to date.

Overview of the SPP Application Process

In March 2012, TSA revised the SPP application to reflect requirements of the FAA Modernization Act enacted in February 2012.[Footnote 20] Among other provisions, the act provides that

-- Not later than 120 days after the date of receipt of an SPP application submitted by an airport operator, the TSA Administrator must approve or deny the application.

-- The TSA Administrator shall approve an application if approval would not (1) compromise security, (2) detrimentally affect the cost- efficiency of the screening of passengers or property at the airport, or (3) detrimentally affect the effectiveness of the screening of passengers or property at the airport.

-- The airport operator shall include as part of its application submission a recommendation as to which private screening company would best serve the security screening and passenger needs of the airport.

-- Within 60 days of a denial TSA must provide the airport operator, as well as the Committee on Commerce, Science, and Transportation of the Senate and the Committee on Homeland Security of the U.S. House of Representatives, a written report that sets forth the findings that served as the basis of the denial, the results of any cost or security analysis conducted in considering the application, and recommendations on how the airport operator can address the reasons for denial.

All commercial airports are eligible to apply to the SPP. To apply, an airport operator must complete the SPP application and submit it to the SPP PMO, as well as to the airport FSD, by mail, fax, or e-mail. As required by the FAA Modernization Act, not later than 120 days after the application is received by TSA, the Administrator must make a final decision on the application. Figure 1 illustrates the SPP application process.

Although TSA provides all airports with the opportunity to apply for participation in the SPP, authority to approve or deny the application resides in the discretion of the TSA Administrator. According to TSA officials, in addition to the cost-efficiency and effectiveness considerations mandated by FAA Modernization Act, there are many other factors that are weighed in considering an airport`s application for SPP participation. For example, the potential impact on the workload of the Office of Information Technology and the potential impact of any upcoming projects at the airport are considered. SPP PMO officials said that by considering all relevant factors, they do not expect to identify a specific piece of information that would definitively deny an application`s approval based on the standards in the FAA Modernization Act. However, in doing so, they hope to ensure that the Administrator has the complete picture and could therefore make a decision using all factors in combination, consistent with the FAA Modernization Act. Nonetheless, factors found to be cost- prohibitive are likely to result in the airport being denied participation in the program.

TSA Implemented Prior Efforts Comparing Private and Federal Screening

In May 2007, TSA awarded a contract to Catapult Consultants to conduct a cost and performance analysis of airports with private screeners versus airports with federal screeners.[Footnote 21] This analysis would be used to assist senior TSA leadership with strategic decisions regarding the degree to which TSA should leverage public/private partnerships in the area of screening services. According to the December 2007 report the contractor issued on its analysis, SPP airports performed at a level equal to or better than non-SPP airports for the four performance measures included in the analysis.[Footnote 22] Following this study, in February 2008, TSA issued a report on a study TSA conducted comparing the cost and performance of screening at SPP and non-SPP airports.[Footnote 23] The study compared performance measures at each of six SPP airports to the non-SPP airports in the same airport category and found that SPP airports generally performed consistently with non-SPP airports in their category for the performance measures included in its analysis.[Footnote 24]

TSA Has Approved 25 SPP Applications; Enhanced Customer Service Was the Most Commonly Cited Advantage of the SPP

SPP Applicants and Participating Airports

Since the inception of the SPP in 2004, 29 airports have applied for participation in the program; 25 airports have been approved, and as we noted earlier in this report, 16 airports are participating in the SPP as of October 2012.[Footnote 25] A detailed timeline and status of each airport application are provided in figure 2 and appendix II.

Nine airports were approved but are not currently participating in the program because they are either (1) in the process of having an SPP contractor procured, (2) were once part of the SPP but ceased screening services when commercial airline service placing the airport under TSA regulation was discontinued, or (3) never transitioned to the SPP because commercial airline service bringing the airport under TSA regulation to these airports was discontinued before private screening services began. Specifically, 6 airports--West Yellowstone Airport, Montana; Orlando Sanford International Airport, Florida; Glacier Park International Airport, Montana; Sacramento International Airport, California; Bert Mooney Airport, Montana; and Bozeman Yellowstone International Airport, Montana--have been approved but are not yet currently participating in the SPP pending TSA`s selection of the screening contractor to provide services at each airport. Two airports--the East 34th Street Heliport, New York, and Gallup Municipal Airport, New Mexico were participating in the SPP, but according to TSA officials, the air carriers servicing these airports discontinued service after the contract was awarded, and thus these airports no longer required TSA screening services. Additionally, Florida Keys Marathon Airport, Florida, was approved for participation in the SPP, but the air carrier servicing the airport discontinued services prior to the start of the screening contract, and accordingly screening services were no longer required.

TSA denied applications from 6 airports--submitted from March 2009 through December 2011. Five of these applications were submitted to TSA before the Administrator announced in January 2011 that the agency would not expand the SPP beyond the then current 16 airports ``unless a clear and substantial advantage to do so emerges in the future.`` The sixth application was submitted for consideration approximately 1 week after the Administrator`s announcement. Prior to the enactment of the FAA Modernization Act in February 2012, 1 of the 6 airports whose application TSA denied re-applied under TSA`s ``clear and substantial advantage`` standard and was approved.[Footnote 26] Following enactment of the FAA Modernization Act, which provided that TSA shall approve an application if approval would not compromise security or detrimentally affect the cost-efficiency or the effectiveness of the screening of passengers or property at the airport, TSA approved the applications of 3 other airports who reapplied. Two of the 6 airports that had been denied never reapplied for participation in the SPP (see figure 2 for additional details). Figure 3 and appendix III show the locations of the 16 airports currently participating in the SPP as well as the 6 airports that TSA recently approved for participation.

As figure 3 shows, 10 of 16 of the airports currently participating in the SPP are smaller, category III and IV airports, with 9 of those located in the western region of the United States.

In recent years, the number of airports applying for participation in the SPP has generally declined. Specifically, from 2004 through 2008, 21 airports applied to the SPP, including the 5 airports that participated in TSA`s SPP pilot program. Since 2009, TSA has received SPP applications from 8 airports.

Reported Advantages and Disadvantages of Joining the SPP

Airport operators we surveyed and interviewed, as well as aviation industry stakeholders (i.e., aviation associations) and TSA officials we interviewed, most commonly cited customer service and staffing flexibility as advantages of participating in the SPP, but also expressed concerns about the SPP transition process and satisfaction with existing TSA screening services as potential disadvantages of participating in the program.[Footnote 27] We surveyed 28 airport operators who had applied to the SPP from its inception in 2004 through April 2012. Twenty-six operators responded. Because all 26 survey respondents were airport operators who have applied to the SPP, these airport operators may be more likely to present positive views of, or what they perceived of, the SPP. In addition, perspectives may also be influenced by whether or not the operators were approved for participation in the SPP at the time the survey was conducted. We also interviewed 6 airport operators that were not included in our survey. Five of these airport operators have not applied for participation in the SPP, and 1 airport operator had applied for participation after our survey was conducted, and therefore was not included as part of our survey.

Our 2012 survey and interviews of airport operators include the following highlights

Advantages to SPP

The advantages most frequently identified by the airport operators that had applied to the SPP and responded to our survey and those we interviewed (including those that had not applied to the SPP) were related to providing better customer service and obtaining flexibility assigning staff. The airport associations most commonly cited obtaining flexibility in assigning staff as an advantage. Because TSA generally remains neutral regarding the SPP, the views of TSA officials expressed are attributed to the individual FSDs we interviewed and do not reflect the views of the agency.

-- Customer service. Sixteen airport operators we surveyed and interviewed reported customer service as an advantage--15 had applied to the SPP and 1 had not.[Footnote 28] Specifically, 14 of 26 airport operators responding to the survey indicated this was a realized or potential advantage to a great or very great extent.[Footnote 29] In addition, 2 of the 6 airport operators we interviewed, 1 of which applied to the SPP, stated that the level of customer service provided by security screeners is particularly important for smaller community- based airports. These airports constitute the majority of the airports participating in the SPP, because passengers who have negative encounters with the screening process generally associate their experiences with the specific airport. Thus, airport officials stated that this might increase the likelihood that the passengers involved will seek alternative modes of transportation or different airports for future travel. Representatives from the three airport associations we interviewed did not identify customer service as an advantage of the SPP. TSA officials stated that federal screeners can and do provide similar levels of customer service and that most commercial airports are content to have a TSA workforce at their airports. TSA also stated that customer service is an important aspect of their work, and that the agency is taking steps to improve customer service in a way that does not jeopardize the agency`s core mission, which is to ensure the security of the traveling public. Specifically, TSA officials said that they have enhanced their performance management processes to better gauge customer service, such as tracking negative contacts received at airports.

-- Staffing flexibility. Fifteen airport operators we surveyed and interviewed--14 had applied to the SPP and 1 had not--and representatives from two aviation industry associations reported that private screening contractors are generally more responsive and flexible than TSA to increasing staffing needs in response to fluctuations in passenger volume at the airport. Specifically, 13 of 26 airport operators responding to our survey cited flexibility in assigning staff as a realized or potential advantage to a great or very great extent of participating in the SPP. Two of the 6 airport operators we interviewed, 1 of which had applied to the SPP, also cited staffing flexibility as an advantage. For example, an airport operator highlighted challenges the airport has faced in adjusting the number of screening staff to accommodate the seasonal changes in passenger volume at his airport. Specifically, the airport operator, a current SPP participant, commented that unlike TSA screeners, private screening contractors are able to staff screeners in split shifts--a work period divided into two or more periods of time, such as morning and evening, with a break of several hours between--thereby enabling them to adjust to the airport`s flight schedule and changes in passenger volume.[Footnote 30] TSA officials disagreed with this view and stated that TSA provides FSDs with discretion to utilize federal screeners in split shifts during the course of the workday, provided that such discretion is exercised as the direct result of operational need. Furthermore, TSA officials stated that all category IV and many category III airports use split shifts. Four of six FSDs we interviewed cited a reduced involvement in human resource management as an advantage to the federal government for participating in the SPP. For example, one FSD said that because TSA oversees the screening operations of SPP airports and FSDs are not involved with deploying and managing screening staff, they are better able to focus on their security oversight functions, including ensuring that proper standard operating procedures are being followed.

-- Cost savings. During our follow-up interviews with survey respondents, 4 airport operators said that participating in the SPP could help alleviate TSA resource constraints and result in cost savings to the federal government because some airports that are currently participating in or applied for participation in the SPP are located in certain rural or high-cost communities where the federal government has difficulty hiring screeners and must utilize federal personnel deployed for temporary assignments, which results in increased costs. An FSD of an SPP airport located in a small, high- cost community we interviewed agreed that the salary offered by TSA made it difficult to fill screening positions at the airport, stating that prior to the airport`s transition to the SPP, TSA had difficulty hiring screeners from the local area, and as a result had to use screeners from its National Deployment Force (NDF), a deployable federal screening workforce, because of the high cost of living in the area.[Footnote 31] To maintain the requisite level of screening services at airports in environments where it is hard to recruit, TSA often uses screeners from its NDF, which TSA stated can be more expensive than SPP screeners because the NDF screeners are compensated on a per diem basis when deployed and incur other costs such as temporary housing expenses.

Disadvantages

Airport operators generally cited few realized or potential disadvantages of participating in the SPP. Six airport operators we surveyed and interviewed cited the discontinuation of federal screening services as a potential disadvantage of participating in the SPP. Specifically, the 4 of 25 survey respondents who had applied to the SPP program cited the discontinuation of federal screening services as a potential disadvantage of participating in the SPP.[Footnote 32] In addition, 2 airport operators who have not applied to the SPP expressed concerns about the potential disruption associated with the transition from TSA screeners to private screeners at their airports, and the associated risk of doing so if the process does not proceed as smoothly as intended.[Footnote 33] One of these airport operators stated that concerns about the transition process-- going from federal screeners to private screeners--is the primary reason the airport has not submitted an application. Further, this airport operator also cited concerns about maintaining screener morale, and hence security, as a major reason for the airport`s decision to not apply to the SPP.[Footnote 34] Officials from the aviation industry associations we interviewed did not cite any realized or potential disadvantages. As noted earlier, TSA generally remains neutral regarding the SPP, and accordingly did not cite disadvantages of participating in the SPP.

Additionally, airport operators from 3 airports that have not applied to the SPP expressed no interest in the SPP, and stated that they are generally satisfied with the level of screening service provided by TSA. Similarly, an Airport Council International-North America (ACI- NA) March 2007 study found that 71 percent of 31 survey respondents were not interested in the SPP, and cited satisfaction with TSA screening services, among other things, for not having any interest in the SPP.[Footnote 35] When asked, representatives from all three aviation industry associations we interviewed either expressed no opinion on the SPP or cited no disadvantages to participating in the SPP. Two of these industry representatives added that the majority of the airports they represent are generally satisfied with the screening services provided by TSA.

TSA Has Developed Application Resources, but Could Provide Guidance for SPP Applicants

TSA has developed some resources to assist applicants; however, it has not provided guidance on its application and approval process to assist airports with applying to the program. As the application process was originally implemented, TSA required that an airport operator interested in applying to the program submit an application stating its intention to opt out of federal screening as well as its reason(s) for wanting to do so. However, in 2011, TSA revised its SPP application to reflect the ``clear and substantial advantage`` standard announced by the Administrator in January 2011. Specifically, TSA requested that the applicant explain how private screening at the airport would provide a clear and substantial advantage to TSA`s security operations.[Footnote 36] At the time, TSA did not provide written guidance to airports to assist them in understanding what would constitute a ``clear and substantial advantage to TSA security operations`` or TSA`s basis for determining whether an airport had established that opting out would present a clear and substantial advantage to TSA security operations. TSA officials told us that they did not issue guidance at the time in conjunction with the new standard because the agency desired to maintain a neutral position on the SPP and did not want to influence an airport`s decision to participate in the program. In the absence of such guidance, SPP officials told us that they were available to provide assistance, if requested, to airports that sought assistance or information on completing their application.

In March 2012, TSA again revised the SPP application in accordance with provisions of the FAA Modernization Act enacted in February 2012. Among other things, the revised application no longer includes the ``clear and substantial advantage`` question, but instead includes questions that request applicants to discuss how participating in the SPP would not compromise security at the airport and to identify potential areas where cost savings or efficiencies may be realized. [Footnote 37] Additionally, in accordance with the FAA Modernization Act, applicants must recommend a contractor that would best serve the security screening and passenger needs of the airport. TSA officials told us that the agency offers potential applicants numerous points of contact and methods with which the applicants can discuss the program before applying to participate. Specifically, applicants can discuss the program with their FSD, the SPP program manager, or their recommended screening contractor. Further, according to TSA officials, once an airport operator submits an application, TSA assigns a program official as a point of contact for the application, and works with the applicant to ensure the application is complete and to keep the applicant informed. TSA also provides general instructions for filling out the SPP application as well as responses to frequently asked questions (FAQ). However, TSA has not issued guidance to assist airports with completing the new application and has not explained to airports how it will evaluate applications given the changes brought about by the new law. Neither the current application instructions nor the FAQs address TSA`s SPP application evaluation process or its basis for determining whether an airport`s entry into SPP would compromise security or affect cost- efficiency and effectiveness.

We interviewed 4 of the 5 airport operators that applied to the SPP since TSA revised its application in the wake of the FAA Modernization Act. Three of the 5 told us that they struggled to answer the application questions related to the cost-efficiency of converting to the SPP because they did not have data on federal screening costs, while the fourth airport operator said that she did not need additional information or guidance to respond to the question. One of the 4 airport operators stated that he needed the cost information to help demonstrate that his airport`s participation in the SPP would not detrimentally affect the cost-efficiency of the screening of passengers or property at the airport and that he believes not presenting this information would be detrimental to his airport`s application. However, TSA officials said that the cost information required to answer the questions is basic cost information that airports should already maintain and that airports do not need to provide this information to TSA because, as part of the application evaluation process, TSA conducts a more detailed cost analysis using historical cost data from SPP and non-SPP airports. TSA officials added that the SPP application and the cost information requested only serve to alert TSA of things it may not be already aware of about the airport. The absence of cost and other information in an individual airport`s application, TSA officials noted, would not materially affect the TSA Administrator`s decision on an SPP application.

Three of the 4 airport operators we interviewed, and whose applications TSA subsequently approved after enactment of the FAA Modernization Act, said that additional guidance would have been helpful in completing the application and determining how TSA evaluates the applications. A representative from 1 of the 3 airports stated that while TSA officials have been more responsive and accessible since enactment of the FAA Modernization Act, the agency has not necessarily been helpful with the application process. Moreover, all 4 airport operators we interviewed told us that TSA did not specifically assign a point of contact when they applied to the program. Rather, all 4 airport operators reported consulting the SPP PMO, their FSD, or their recommended contractor because they needed information on such issues as screening cost, the list of current SPP contractors, TSA screener staffing levels, and examples of additional information they should provide TSA because they could not answer some of the application questions without this information. Specifically, 1 of the 4 airport operators reported contacting the FSD to request assistance with completing the application, while 2 of the four said they did not because FSDs generally are not knowledgeable about the program or are able to provide only general as opposed to detailed information about the application process. Instead of contacting their FSDs, these 2 airport operators told us that they contacted the SPP PMO and stated that the office were helpful in providing general information, such as a list of current SPP contractors, but not screening cost or other specific application information that would help the airports demonstrate whether the use of private screeners would compromise security or detrimentally affect the cost-efficiency or effectiveness of the screening of passengers or property at the airport. Another airport operator who reported contacting the SPP PMO stated that she learned about TSA`s SPP selection criteria and processes in the course of her discussions with one of the SPP managers with whom she had developed a working relationship over the years, and added that had she not contacted this particular manager, she would not have obtained this information because TSA does not publish the information for other airports that may be interested in obtaining the information. Three of the 4 airport operators who told us they sought information to complete their application from their recommended contractor as advised by TSA stated that the contractors told them they did not have the necessary cost information to assist the airports with responding to the application questions related to the cost-efficiency of converting to the SPP.

Following enactment of the FAA Modernization Act, TSA officials initially stated that application guidance is not needed because the ``clear and substantial`` basis for joining the SPP has been eliminated and responses to the two new application questions related to cost- efficiency and effectiveness are optional responses. However, the Assistant Administrator for the Office of Security Operations now agrees that providing additional high- level guidance on the kind of information TSA considers during the application review phase would be helpful to SPP applicants. TSA SPP officials also stated that they routinely talk about the SPP at industry briefings and that they have done a good job of explaining the new application to industry. However, as of September 2012, representatives of all three aviation industry associations we interviewed told us that TSA has not provided any information on the SPP to their association since enactment of the FAA Modernization Act in February 2012. Additionally, representatives of two of the three aviation industry associations said that providing guidance or information on the criteria TSA uses to evaluate applications would be helpful to their members, while a representative from the third aviation association that represents domestic and international airline carriers said that its members would appreciate any basic information on the SPP. In interviews we conducted prior to the enactment of the FAA Modernization Act, these same aviation industry representatives told us that the absence of guidance provided by TSA is a barrier to applying to the program. They added that most airports do not want to invest in preparing an application when they are unsure as to how it would be evaluated by TSA.

TSA has approved all applications submitted since enactment of the FAA Modernization Act; however, it is hard to determine how many more airports, if any, would have applied to the program had TSA provided application guidance and information to improve transparency of the SPP application process. In the absence of such application guidance and information, it will be difficult for more airport officials to evaluate whether their airports are good candidates for the SPP or determine what criteria TSA uses to accept and approve airports` SPP applications. Further, airports may be missing opportunities to provide TSA with cost and other information that TSA would find useful in reviewing airport applications. According to Standards for Internal Control in the Federal Government, internal control and all transactions and other significant events need to be clearly documented, and the documentation should be readily available for examination.[Footnote 38] The documentation should appear in management directives, administrative policies, or operating manuals and may be in paper or electronic form. Clear guidance for applying to the SPP could improve the transparency of the SPP application process and help ensure that the existing application process is implemented in a consistent and uniform manner.

TSA Has Measures to Assess Screener Performance, but Enhanced Processes for Evaluating Screener Performance Could Be Beneficial

TSA improved its set of screener performance measures in 2012 by adding measures that address passenger satisfaction, thereby ensuring that the measures address all aspects of the agency`s airport screening strategic goals and mission. However, a mechanism to monitor private versus federal screener performance could help TSA to routinely ensure that the level of screening services and protection provided at SPP airports continues to be conducted at acceptable levels provided at non-SPP airports, and could help inform TSA managers when making decisions regarding the future of the SPP, such as whether to expand the program to more non-SPP airports. While we found differences in screener performance between SPP and non-SPP airports, those differences cannot be entirely attributed to the use of either private or federal screeners.

Performance between SPP and Non-SPP Airports Varies for Some Measures, but Differences Cannot Be Entirely Attributed to the Use of Private or Federal Screeners

We analyzed screener performance data for four measures and found that while there are differences in performance between SPP and non-SPP airports, those differences cannot be exclusively attributed to the use of either federal or private screeners. We selected these measures primarily based on our review of previous studies that compared screener performance of SPP and non-SPP airports as well as on our interviews with aviation security subject matter experts, including TSA`s FSDs, SPP contractors, and airport and aviation industry stakeholders. We also selected performance measures for which TSA has, for the most part, consistently and systematically collected data from fiscal year 2009 through 2011.[Footnote 39] The measures we selected to compare screener performance at SPP and non-SPP airports are TIP detection rates, recertification pass rates, Aviation Security Assessment Program (ASAP) test results, and Presence, Advisement, Communication, and Execution (PACE) evaluation results (see table 1). For each of these four measures, we compared the performance of each of the 16 SPP airports with the average performance for each airport`s category (X, I, II, III, or IV), as well as the national performance averages for all airports for fiscal years 2009 through 2011.

On the basis of our analyses, we found that, generally, certain SPP airports performed slightly above the airport category and national averages for some measures, while others performed slightly below. For example, SPP airports performed above their respective airport category averages for recertification pass rates in the majority of instances, while the majority of SPP airports that took PACE evaluations in 2011 performed below their airport category averages on their PACE evaluations.[Footnote 40] For TIP detection rates, SPP airports performed above their respective airport category averages in about half of the instances. DHS deemed the details of our analyses of the four performance measures we used for comparing SPP with non-SPP screener performance as classified or sensitive security information; thus, these details are not included in this report.[Footnote 41]

The differences we observed in private and federal screener performance cannot be entirely attributed to the type of screeners (private or federal) at an airport, because, according to TSA officials and other subject matter experts we interviewed, many factors, some of which cannot be controlled for, affect screener performance. These factors include, but are not limited to, checkpoint layout, airline schedules, seasonal changes in travel volume, and type of traveler. For example, TSA officials told us that the type of traveler experienced by an airport can affect the average wait time at an airport. Airports located in areas near tourist locations, for example, may experience higher volumes of first-time and infrequent travelers, as opposed to business travelers that fly more frequently. Infrequent travelers are more likely to bring prohibited items through the checkpoint because they are unfamiliar with TSA`s security protocols, a fact that can result in more bags needing to be searched and, consequently, longer wait times. Accordingly, while there may be differences in performance between an airport near a tourist location and an airport not located near a tourist location, those differences may be attributed to the type of passenger and not the use of either federal or private screeners at these airports. However, while differences in performance cannot be entirely attributed to the type of screener, differences and changes over time may still be of interest to TSA managers and may inform decision making regarding the future use of private contractor screeners, which we discuss later.

TSA collects data on several other performance measures but, for various reasons, they cannot be used to compare private and federal screener performance for the purposes of our review. For example, we do not present passenger wait time data because we found that TSA`s policy for collecting wait times changed during the time period of our analyses and that these data were not collected in a consistent manner across all airports. We also considered reviewing human capital measures such as attrition, absenteeism, and injury rates, but did not analyze these data because TSA`s Office of Human Capital does not collect these data for SPP airports. While the contractors collect and report this information to TSA, TSA does not validate the accuracy of the self-reported data nor does it require contractors to use the same human capital measures as TSA, and accordingly, differences may exist in how the metrics are defined and how the data are collected. Therefore, TSA cannot guarantee that a comparison of SPP and non-SPP airports on these human capital metrics would be an equal comparison. In appendix IV, we discuss these two variables as well as two other variables occasionally cited by the airport officials and aviation stakeholders we interviewed as possible measures for comparing federal and private screening and the reasons we did not use them to compare private and federal screener performance.

TSA Recently Improved Its Screener Performance Measures to Ensure the Measures Address Strategic Goals and Mission

Beginning in April 2012, TSA introduced a new set of performance measures to assess screener performance at both SPP and non-SPP airports in its OSO Executive Scorecard (Scorecard). OSO officials told us that they provide the Scorecard to FSDs every 2 weeks to assist the FSDs with tracking performance against stated goals and with determining how performance of the airports under their jurisdiction compares with national averages. According to TSA, the 10 measures now used in the Scorecard were selected based on input from FSDs and regional directors on the performance measures that most adequately reflect screener and airport performance. Prior to the Scorecard, from 2006 through April 2012, FSDs used three performance measures in the Management Objective Report (MOR) to assess screener and airport performance (see table 2). Further, TSA improved upon the set of measures it uses to assess screener performance by adding measures to the Scorecard that addressed other non-security- related TSA priorities, such as passenger satisfaction. Specifically, the Scorecard includes passenger satisfaction measures, such as the number of negative and positive customer contacts made to the TSA Contact Center through e-mails or phone calls per 100,000 passengers screened through the airport, which were not previously included in the MOR. [Footnote 42] By adding measures related to passenger satisfaction to the Scorecard, TSA helped to ensure balance--that is, addressing a variety of agency goals--in the set of performance measures the agency uses to assess screener performance, which helps to ensure that performance measurement efforts are not overemphasizing one or two priorities at the expense of others. Details on our assessment of the MOR and Scorecard are provided in appendix V. While many of the measures used to assess screener performance are included in the Scorecard, several are not, but are available to TSA officials through other reports and databases. For example, TSA officials are able to review reports on their passenger throughput, wait times, and covert test results-- information that is not included in the Scorecard.

Mechanisms to Monitor Private Screener Performance Separately from Federal Screeners Could Benefit TSA

TSA does not currently monitor private screener performance separately from federal screener performance or conduct regular reviews comparing the performance of SPP and non-SPP airports. As previously noted, TSA has conducted or commissioned prior reports comparing the cost and performance of SPP and non-SPP airports. For example, in 2004 and 2007, TSA commissioned reports prepared by private consultants, while in 2008 the agency issued its own report comparing the performance of SPP and non-SPP airports.[Footnote 43] Generally, these reports found that SPP airports performed at a level equal to or better than non-SPP airports.[Footnote 44] However, TSA officials stated that they do not plan to conduct similar analyses in the future, and instead, they are using across-the-board mechanisms of both private and federal screeners, such as the Scorecard, to assess screener performance across all commercial airports.

In addition to using the Scorecard, TSA conducts monthly contractor performance management reviews (PMR) at each SPP airport to assess the SPP contractor`s performance against the standards set in each SPP contract.[Footnote 45] The PMRs include 10 performance measures, including some of the same measures included in the Scorecard, such as TIP detection rates and recertification pass rates, for which TSA establishes acceptable quality levels of performance.[Footnote 46] Failure to meet the acceptable quality levels of performance can result in corrective actions or termination of the contract.[Footnote 47] For example, according to TSA officials, TSA developed reports for several of the airports that included corrective actions related to the protection of sensitive security information.

However, the Scorecard and PMR do not provide a complete picture of screener performance at SPP airports because, while both mechanisms provide a snapshot of private screener performance at each SPP airport, this information on screener performance is not summarized for the SPP as a whole or across years, which makes it difficult to identify changes in private screener performance. For example, an airport`s Scorecard shows the performance of that airport during a 2- week reporting period, as well as for the year to date, in comparison with the airport`s goal for each of the performance measures. However, it does not show that airport`s performance in comparison with that of others in its airport category, which TSA officials explained is important when assessing screener performance. Likewise, the PMRs present information on an SPP contractor`s performance against the standards in their contract during a particular month. With the exception of TIP detection rates and recertification pass rates, the PMRs do not compare an SPP airport`s performance against other airports or, for example, its airport category average, which TSA officials stated is important when assessing screener performance. [Footnote 48] TSA stores paper copies of the results from the performance reviews, but it does not transfer the information into an electronic system or format that would allow the agency to easily review SPP performance over time. During the course of our audit work, TSA officials informed us that they have identified this as an area needing improvement and plan to introduce a new tool to collect and consolidate this information in fiscal year 2013, but were unable to provide specific information on the format of this tool and how it will be used. Further, neither the Scorecard nor the PMR provides information on performance in prior years nor controls for variables that TSA officials explained to us are important when comparing private and federal screener performance, such as the type of X-ray machine used for TIP detection rates.

Monitoring private screener performance in comparison with federal screener performance is consistent with the statutory requirement that TSA enter into a contract with a private screening company only if the Administrator determines and certifies to Congress that the level of screening services and protection provided at an airport under a contract will be equal to or greater than the level that would be provided at the airport by federal government personnel.[Footnote 49] Further, according to TSA guidance on the SPP, one of TSA`s major goals for the SPP is that private screeners must perform at the same or better level as federal screeners. A mechanism to monitor private versus federal screener performance would better position TSA to know whether the level of screening services and protection provided at SPP airports continues to be equal to or greater than the level provided at non-SPP airports. TSA officials stated that it is not TSA`s goal to ensure that SPP airports continue to perform at levels equal to or greater than non-SPP airports, but to ensure that all airports operate at their optimal level, which they monitor using across-the-board mechanisms, such as the Scorecard. However, monitoring private versus federal screener performance could also help TSA to identify positive or negative trends in SPP performance that could lead to improvements in the program and TSA`s monitoring of SPP airports in general, and inform decision-making regarding potential future expansion of the SPP.

Conclusions

TSA faces a daunting task in ensuring that a screening workforce is in place to consistently implement security protocols across the nation`s commercial airports while facilitating passenger travel. Questions about the performance of private screeners compared with federal screeners, recently enacted statutory provisions, and changes to the program`s application and approval process underscore the need for TSA to ensure that the program`s application requirements are clearly defined and consistently applied so that aviation stakeholders have a full and fair opportunity to participate in the program. Thus, a well- defined and clearly documented application guideline that states (1) the criteria and process that TSA is using to assess airport`s participation in the SPP, (2) how TSA will obtain and analyze cost information regarding screening cost-efficiency and effectiveness and the implications of not responding to related application questions, and (3) specific examples of additional information airports should consider providing to TSA to help assess airports` suitability for SPP could benefit TSA. Specifically, guidelines could help alleviate airports` uncertainty about the application process and better inform TSA to determine whether to approve an airport`s SPP application.

It is also incumbent on TSA to be capable of determining if airports participating in the program are performing at a level that is equal to or greater than the level of security that would be provided by federal screeners at the airports through regular monitoring and reporting. Although not a prerequisite for approving an application for participation in the SPP, TSA must certify to Congress that the level of screening services and protection provided by a private screening contractor will be equal to or greater than the level that would be provided at the airport by federal government personnel before entering into a contract with a private screening company. While TSA regularly tracks screener performance at all airports and reevaluates the measures it uses to assess this performance, TSA has not conducted regular reviews comparing private and federal screener performance and does not have plans to do so. Regular comparison reviews would enable TSA to know whether the level of screening services provided by private screening contractors is equal to or greater than the level provided at non-SPP airports. These reviews could also assist TSA in identifying performance changes that could lead to improvements in the program and inform decision making regarding potential expansion of the SPP.

Recommendations for Executive Action

To improve TSA`s SPP application process and to inform decisions regarding the future of the SPP, we recommend that the Secretary of the Department of Homeland Security direct the Administrator of TSA to take the following two actions

-- develop guidance that clearly (1) states the criteria and process that TSA is using to assess whether participation in the SPP would compromise security or detrimentally affect the cost- efficiency or the effectiveness of the screening of passengers or property at the airport; (2) states how TSA will obtain and analyze cost information regarding screening cost-efficiency and effectiveness and the implications of not responding to the related application questions; and (3) provides specific examples of additional information airports should consider providing to TSA to help assess an airport`s suitability for SPP, and

-- develop a mechanism to regularly monitor private versus federal screener performance.

Agency Comments and Our Evaluation

We requested comments on a draft of the sensitive version of this report from TSA. On November 7, 2012, DHS provided written comments, which are reprinted in appendix VI and provided technical comments, which we incorporated as appropriate. DHS generally concurred with our two recommendations and described actions planned to address them. Specifically,

-- DHS stated that TSA will provide as much information as is prudent on how the agency would evaluate if an airport`s participation in the SPP would compromise security or detrimentally affect the cost- efficiency or the effectiveness of the screening of passengers or property at the airport. Further, DHS stated that TSA will provide general categories of information in the SPP application guidance it plans to issue and will continually review the guidance to ensure that airports are comfortable with the SPP application process and understand how all the information provided will be used to evaluate their application. TSA expects to post an overview of the SPP application process to the agency`s website by November 30, 2012, that would specify details on the data it will use to assess applications and discuss its cost-estimating methodology and definition of cost efficiency. We believe that these are beneficial steps that would address our recommendation once adopted, and help address stakeholder concerns about the transparency of the SPP application process.

-- DHS stated that starting in the first quarter of fiscal year 2013, TSA will produce semi-annual reports that will include an evaluation of SPP airport performance against the performance of TSA airports as a whole, as well as performance against each SPP airport category. Additionally, DHS noted that TSA is in the initial planning phase of deploying an electronic data collection system to facilitate systematic collection and reporting of SPP data, as well as TSA oversight of SPP contractor activities. Deployment of the electronic data collection system is targeted for the latter part of fiscal year 2013. Once implemented, these new reporting mechanisms will address our recommendation by facilitating TSA`s efforts to assess private versus federal screener performance.

We are sending copies of this report to the Secretary of Homeland Security, the TSA Administrator, the House Infrastructure and Transportation Committee, and other interested parties. In addition, the report is available at no charge on the GAO web- site at [hyperlink, http//www.gao.gov].

If you or your staff have any questions concerning this report, please contact me at (202) 512-4379 or at [email protected] Contact points for our Offices of Congressional Relations and Public Affairs may be found on that last page of this report. GAO staff who made major contributions to this report are listed in appendix VII.

Sincerely Yours,

Signed by

Stephen M. Lord Director, Homeland Security and Justice Issues

Copyright 2012 Congressional Quarterly, Inc. All Rights Reserved.