The much-awaited results from the Fed’s latest stress tests on large banks, formally known as the Comprehensive Capital Analysis and Review 2012 (CCAR), must be put into context around the effort’s massive dependencies on models and assumptions used to determine each firm’s capital situation.
The Fed’s report implies much more precision than lies within their analysis. Upon close inspection of the methodology for generating the stress scenario projections, it is clear that much work remains for the Fed to develop a robust stress test capability. Conceptually, the approach to apply a consistent analytical methodology and set of assumptions against all banks is sound and the Fed has made significant strides in developing this capability since their first try at this in 2009 with the Supervisory Capital Assessment Program (SCAP). Nevertheless, these latest stress tests are fraught with the same kind of model risk that has historically plagued banks.
The first issue relates to the Fed’s approach to assessing losses on bank accrual portfolios. An industry-level model is developed against which all bank portfolios are measured. While this establishes a common set of risk factors describing the various loan types, it tends to wash out bank-specific effects. That is, it tends to make riskier accrual portfolios appear less risky than they are and vice versa. This occurs by aggregating data across the industry rather than building separate bank-specific models, which would be even more resource intensive than the massive stress test effort already entails. Technical workarounds to this problem are possible, but are beyond the scope of this discussion. A second issue with the stress test relates to its assessment of operational risk and mortgage repurchase losses. Providing reliable estimates of these risks is notoriously difficult as even the Fed admits in its methodology document. Obtaining statistically reliable estimates of both risks due to the lumpiness of such events and their low frequency, high severity nature is extremely difficult. A third issue with the stress test is what it doesn’t include. Surprisingly, the stress test does not incorporate what many claim contributed to the financial crisis of 2008 in the first place, namely liquidity risk. The fact that liquidity risk assessment does not appear in the CCAR is puzzling given the amount of discussion it has received since the crisis and the focus on including separate capital charges for liquidity risk in Basel III. A fourth issue with CCAR is its over-reliance on a single stress path. The stress scenario selected by the Fed features 25 macroeconomic factors tracked over the stress period. Nowhere in the methodology is it ever described what the sensitivity is of various positions, revenues and expenses to this single scenario. For instance, in assessing counterparty risk exposure, the Fed has to make assumptions regarding asset correlations that drive obligor defaults. If we have learned anything about models used before the crisis, it is that correlations are not static. How are we to know whether credit losses of some part of the portfolio, for example, are overly sensitive to one or more of these factors, thereby introducing instability into the estimates? Moreover, the Fed acknowledges that it conducted an independent model assessment by using economists from across the Federal Reserve System. For such a critically important exercise, the Fed ought to have engaged outside scholars as well as economists from other regulatory agencies and industry practitioners with appropriate analytical and banking subject matter expertise. One of the strengths of CCAR is also its weakness; namely its complexity. The methodology is a testament to large-scale empirical modeling efforts that can easily engender the kind of overconfidence in results that such efforts met with before the crisis.
In light of these limitations, what should we make of the CCAR results this week? The first takeaway is that the Fed has at least made a heroic attempt to provide a comprehensive picture of large bank vulnerabilities to significant stress events. It isn’t clear whether the Fed is measuring the right stress test or even what that means. But it does provide a common benchmark against a single stress event to compare bank performance in the spirit of Vikram Pandit’s bank risk model proposal. But investors and others closely following the numbers should realize that the capital results generated by CCAR are subject to variability due to inherent model sensitivities. The effort remains a black box to all but the Fed and until the veil is lifted on the models and assumptions, its results should be viewed with a healthy dose of skepticism.