Measuring Success in Policy Not as Easy as it Seems

Jun 11, 2015

Andrew Downs, Director of the Mike Downs Center for Indiana Politics at IPFW.
Credit Courtesy / Andrew Downs

The Indiana General Assembly completed its work right at the mandated deadline of April 29th and most of the legislation that was passed will go into effect on July 1st.  While people might think of the legislative process as being over at this point, the public policy process is not.  An often forgotten part of the process is determining whether or not a policy has worked.

In a perfect world, when policies are being developed, the alternatives that are being considered are evaluated before one of them is adopted.  One way this happens in Indiana is through fiscal impact statements.  These statements give legislators (and anyone else who reads them) an idea of what the proposed law is likely to cost and how much revenue is likely to be generated. 

The fiscal impact statements provide a simple way to look at the efficiency of the proposed policy.  In other words, the statements let us know if the revenues exceed the costs. 

This year there were several pieces of legislation about alcohol.  Three that became law were House Bill 1053, Senate Bill 113, and Senate Bill 297

  • HB 1053 allows any combination of a microbrewery, farm winery, and an artisan distillery in the same building to sell their products from the same bar (“without a structure separating the service of wine, the service of beer, and the service of liquor“). 
  • SB 113 eases restrictions on the direct sale of wine.
  • SB 297 increases the number of barrels of beer that a microbrewer can produce from 30,000 barrels to 90,000. 

It is a safe bet that a small percentage of Hoosiers read any of these bills and an even safer bet that an even smaller percentage read the fiscal impact statements.  There are two themes in these statements. 

The fiscal impact statement for SB 113 and the fiscal impact statement for SB 297 state that these laws are unlikely to have much of an impact on state or local expenditures because the duties fall within the existing duties of the Indiana Alcohol and Tobacco Commission.[1]  The fiscal impact statements for all three laws state that it is difficult to determine with certainty, but that the laws are likely to result in no increase or a small increase in tax revenue.  If efficiency (benefits exceeding costs) is all that matters, then these laws pass the test.  The argument now can be about whether or not the ratio of benefits to costs is high enough to make adopting the policies worthwhile. 

Another way to evaluate policies is by asking if they have achieved the goals they were to achieve (effectiveness).  This can be more difficult to determine than efficiency because the goals of policies are not always defined clearly.  Even when they are defined clearly, it can be difficult to gather information that will answer the question. 

In the case of these three laws two goals appear to be to reduce restrictions and allow for the growth of businesses.  The first goal is relatively easy to measure.  The second goal is more complicated.  If no breweries, wineries, or distilleries in Indiana see growth, then did the policy fail?  Perhaps not.  It could be that the market shrank.  It will not be possible to include every variable that could influence the growth and shrinkage of businesses. 

One method of evaluating policies that is not used that often is whether or not the policy will extend or restrict privacy and/or individual rights (liberty / freedom).  This is an evaluative criteria that is important to everyone, but it can be much more difficult than the others discussed so far to use. 

One way to think of this criteria is to ask a question: where do your rights end and mine begin?  The way this criteria might be used in the case of these three laws would be to ask if making it easier to purchase alcohol will create a danger that society cannot, or should not bear. 

Evaluating policies can be difficult.  Evaluations often do not give definitive answers to the questions being asked.  The result is that human subjectivity will play a role in whether policies are labeled successes or failures and whether policies will continue, be modified, or terminated.

[1] The fiscal impact statement for HB 1053 does not discuss the effect on state or local expenditures.

Andrew Downs is Director of the Mike Downs Center for Indiana Politics at IPFW.

Opinions expressed in this column are those of the individual writer and do not necessarily reflect the opinions of the staff, management or board of Northeast Indiana Public Radio. If you want to join the conversation, head over to our Facebook page and comment on the post featuring this column.