Chapter X. – Transaction Testing

Chapter X. – Transaction Testing
Reasons for Conducting Transaction Testing
Testing Methods
Judgmental
Statistical
Sampling Concepts
Population Selection
Sample Design and Selection
Sample Review
Evaluation and Interpretation of Results
Summary of Examination Goals – Transaction Testing

X. Transaction Testing

Transaction testing generally refers to the testing of individual loans and is also known as account testing, account sampling, or transaction-level testing. Transaction testing is generally performed as part of each full scope examination of a bank engaged in credit card lending and is sometimes incorporated into target examinations or visitations when helpful in analyzing the areas slated for review. According to the Expanded Guidance for Evaluating Subprime Lending Programs, transaction testing should be completed at each regularly-scheduled examination of banks engaged in subprime lending. Transaction testing is one of the best techniques to unearth the true quality of card portfolios and loan administration practices.

Reasons for Conducting Transaction Testing
Transaction testing is used to determine whether:

• Individual loans adhere to policy, underwriting, risk selection, and pricing standards.
• Management, board, and regulatory reports are accurate and timely.
• Loan accounting and servicing meet appropriate standards, including those for account management.
• Key risk controls and control processes are adequate and functioning as intended. Some examinations have revealed that even when written policies and procedures appear satisfactory, system settings or other devices might be allowing activity that is contrary to the policies and/or that does not meet applicable guidance and laws.
• Roll-rates and other loss forecasting methods used to determine loss allowance levels are accurate and reliable.
• Lending practices exist that may appear unsafe, unsound, abusive, or unfair.

Its results may also aid examiners in determining adverse classifications. The findings of transaction-level testing are incorporated into examination conclusions regarding overall asset quality, the adequacy of loss allowances and capital levels, and the adequacy of risk management practices.

back to top

Testing Methods
The FDIC generally does not require the use of any specific sampling methodology in its safety and soundness examinations of banks engaged in credit card lending. Rather, the type and extent of sampling used is generally left to the discretion of the examiner-in-charge (EIC). The EIC's use of common sense and judgment are critical in determining the focus and extent of testing. To decide on the type of sampling to use, examiners consider the quantity, quality, and nature of the population to be reviewed; the bank's risk management systems; the bank's appetite for risk; the objectives and benefits of the different sampling methods; and the purpose and objective of the sample. Transaction testing is most often categorized into two methods: judgmental or statistical. A brief discussion of each of these methods follows.

Judgmental
The judgmental method is often used during examinations and provides for sampling without statistical measurement. With this method, examiners identify the bank's areas of greatest risk exposure and select items for review using sound judgment and knowledge of a bank's policies, controls, and systems. This method allows examiners to review an identified percentage (coverage) of a specific population. Although examiners cannot statistically relate the results to the entire population, they can identify specific exceptions.

Statistical
Statistical sampling can also be considered and is effective when testing portfolios of homogeneous accounts. Statistical methods are more complex than judgmental methods, but the benefit is the ability to quantify the results and state with a statistically valid confidence that the results are reliable. The complexity of the two types of statistical sampling (proportional and numerical) generally makes them fall outside the scope of this manual. However, brief comments on statistical methods are offered.

With proportional sampling, the population to be sampled is defined by dollar amount. Proportional sampling can be useful to evaluate the quality of a loan portfolio because of the effect larger dollar items can have on asset quality.

Numerical sampling is appropriate for cases in which the frequency of errors, exceptions, or other feature of interest is of primary concern and the dollar amount of the exception is not considered relevant. The sample population is defined by the number of items and is mainly used to reveal the presence (or absence) of a defined characteristic in a portfolio of items with similar characteristics. Each population account has the same probability of selection as any other.

back to top

Sampling Concepts
Key attributes of each sampling concept are discussed in the next sections and mainly focus on judgmental sampling. But, both judgmental and statistical methods entail the same concepts:

• Population selection.
• Sample design and selection.
• Sample review.
• Evaluation and interpretation of results.

Population Selection
Populations are usually gathered via data queries that may be requested during the pre-examination process. Additional population queries are sometimes asked for during the examination when areas of concern are identified (possibly through policy review or other mechanisms) that are not already queried or if prior queries need refinement.

When selecting the populations for review, examiners should consider that querying a broad category (for example, all delinquent loans) can result in a very large population and in a query that may be difficult to work with and/or that may be ineffective to serve the testing objective. When possible, examiners should fine-tune query requests to retrieve only a segment or segments of a large portfolio, where the segment(s) would be reflective of the features of interest. Some examples of how to potentially sample for specific objectives are provided in this table:

Examples of some common populations are housed in the sample pre-examination request list in Appendix A. The list is for example purposes only and should be customized to reflect queries that are risk-focused for the particular circumstances of the examination. For example, if the prior examination identified concerns about certain practices, examiners would likely want to include queries designed to test whether or not management has addressed those concerns.

Sample Design and Selection
The size and composition of the sample drawn from the population should be commensurate with the risk characteristics of the population being tested. The size of sample group should also be sufficient to reach a supportable conclusion and should not be out of proportion to the overall population size that the sample is being drawn from.

In general, accounts should be randomly selected from each data query. Examiners sometimes conduct data sorts to get a feel for the content of the query (such as range of receivable balances, whether any duplicate accounts are included, and so forth). The data sorts help to make sure that the results of the query appear consistent with the query selection criteria and to make sure that there does not appear to be a mistake with the bank's query process. When conducting these data sorts, certain accounts of potential concern may become evident. While examiners should not ignore accounts of potential concern, they should not skew their sample towards these accounts either. Rather, in addition to pulling in some of potentially concerning accounts, examiners should ensure that a random sample of the remaining population is also selected. The random sample may be developed by extracting every n^th account (for example, every 20^th account in the data file) or some other method of the examiner's choosing.

Attempts by management to substantially influence or control the sampling selection process should be brought to the EIC's attention, and management should generally not select the sample. However, examiners should consider management's internal transaction testing practices when determining the size of the sample for the examination. For example, if management conducts timely, comprehensive and thorough transaction testing as part of its audit procedures and the bank is generally well-run, examination sample sizes may be reduced absent any other influential factors (such as rapid growth, soaring delinquency rates, and so forth). However, the sample size must still remain sufficient to support conclusions drawn.

Sample Review
After the sample is identified, examiners begin review of the accounts in the sample. Many banks provide examiners with controlled (such as read-only) access to terminals. With terminal access, examiners are granted password access and review the account's informational screens. It may be necessary to review consecutive months of activity on an account to see how certain transactions impact the account. Generally, the computer has printer access such that the examiner is able to print documents when necessary to support examination findings.

Whether the review utilizes hard copy or electronic information, examiners must be familiar with descriptions of transaction codes, internal and external status codes, reason codes, and other system settings that may assist them in efficiently and effectively reviewing the accounts. Lists of such items are usually requested during the pre-examination process.

While the objective of a sample is to review accounts for a specific characteristic (or feature of interest), examiners should remain alert for any other unusual activity or characteristics in regards to the account. Examiners should also be alert for potential discriminatory, unfair, deceptive, abusive, or predatory lending practices.

Evaluation and Interpretation of Results
If exceptions are absent, no further evaluation is required. Rather, examination workpapers should be prepared to state that no exceptions were identified in the sampled pool. But, when exceptions are found, examiners should further analyze and evaluate the exceptions in an attempt to determine the root causes of the exceptions and whether the exceptions are isolated occurrences or a pattern or practice. Some possibilities to explain an exception could be:

• Bank personnel's inexperience, lack of training, or inadequate policy knowledge.
• An intentional disregard for policies, procedures, laws or other guidance.
• Improper system settings.

However, many other explanations may exist. If a tested sample reveals numerous exceptions, the sample should be expanded until the actual quality of the population can be reasonably estimated. Results from judgmental sampling cannot be projected beyond the accounts sampled. However, exceptions may suggest a larger problem and should be considered when evaluating the quality of the population from which the sample was drawn.

back to top

Summary of Examination Goals – Transaction Testing
Account-level testing is intended to help gauge whether management adheres to policies as well as regulatory and other guidance; whether its risk management practices are acceptable; and whether there are any factors influencing asset quality that are not readily apparent during portfolio-level review. When the bank's profile reflects elevated risk (such as subprime lending), an increase in delinquency or loss rates, new lines of business, new acquisition channels, rapid growth, or inadequate internal testing or audits, account-level testing becomes an even more important and integral part of the examination process. For account testing to be effective, examiners must carefully determine the populations to be sampled, prudently select the accounts to be sampled from those populations, thoroughly review the chosen accounts, and ensure that conclusions are well-thought out and fully-supported. Examiners then incorporate the findings into the examination conclusions regarding overall asset quality, the adequacy of loss allowances and capital levels, and the adequacy of risk management practices.

back to top