By David Elliott
Over the past decade I’ve been increasingly exposed to challenge funds (matching grants allocated on a competitive basis). I’ve managed one, I’ve evaluated several, and even designed a couple. As time has passed I’ve become increasingly concerned about a growing disconnect between the continuing lack of proven effectiveness and results of challenge funds, and the seemingly unquestioning propagation of challenge funds by certain donors (DFID in particular).
I was inspired to write something here having read “The impact of direct support to R&D and innovation in firms: a compendium of evidence on the effectiveness of innovation policy intervention”[1]. This paper comments on public funding instruments – referred to collectively here as challenge funds – to private firms in support of promoting firm innovation and upgrading.
The general evaluation logic of challenge funds
The paper outlines the following evaluation logic and findings for challenge funds. It represents this in terms of the concept of “additionality” at different levels:
- Input additionality: market failures (e.g. information / coordination) lead to an under-investment in innovation by firms, which dampens competitiveness and growth, undermining wider economic and social performance at a more macro level. Public funds seek to correct this market failure through using grants to encourage firms to invest more, more quickly, or into more risky propositions, than they otherwise would have done. In other words, grants can generate input additionality. The wider evidence shows that input additionality of challenge funds is strong (i.e. recipients confirm that the grant encouraged them to do something more, more quickly or different to what they would otherwise have done), but that this means little by itself due to significant respondent bias – they say nice things in the expectation of more grants; and the wider evidence says they generally get them too…!
- Output additionality: investments should yield returns in getting new products or services to new markets. However, having invested more because of the input additionality of the grant support, intuitively firms should expect to realise additional output (i.e. more revenue, more quickly, or for longer if “first mover” status is achieved). However, (from wider experience of challenge funds investing in innovation at least) there are hardly any credible / robust evaluations on the impact on outputs generally, and fewer still that even attempt to measure and attribute additionality to the challenge fund grant.
- Behavioural additionality: initial thinking was that this didn’t matter and/or would be automatic. Assuming success, rational firms would review cost-benefit returns and realise that the higher initial investments yielded higher than expected returns. That, being rational, this would “demonstrate” to them that they should, in future, modify their attitude and responsiveness to risk and investment. Such attitude and behaviour change is critical to sustain progress. Without it, the ‘innovation frontier’ will simply revert back to where it was at the time of initial public intervention, sparking a call for more public action. A cycle of dependency evolves here. The study of behavioural additionality has emerged in response to this dynamic. Unless it’s taken seriously, and unless it’s realised in practice, challenge grants cannot be deemed ‘transformational’ but, rather, national governments would need to adopt a more permanent public good funding focus to innovation. From a review of wider experience, only one evaluation globally was found to have considered this issue explicitly and concluded that grants (alone and discrete) did not realise any behavioural additionality (i.e. they have not proven transformational). However, when used in a more programmatic context, Cunningham et al. find that grants can be part of a process that can lead to lasting change, providing they offer:
- Specialisation and focus: the market failure challenge of the challenge fund should be clearly defined and contextualised;
- Flexibility: both in terms of when support is provided, the degree of intensity of support, and the nature of the support being offered;
- Engagement beyond the firm: both in terms of stimulating ‘spillovers’ to occur in practice, and in recognition that it’s rare for firms to be able to internalise complete responses to market failures; and
- Tighter measurement: if the rationale for intervention is based on market failure, then it’s incumbent on the programme to measure from the point of grant intervention to the higher level objective of market level impact.
Challenge funds and DFID
DFID ‘does’ challenge funds. It likes them. It’s funding more and more of them, in more and more areas, with ever growing volumes of funds (hundreds of millions of GBP). At one level, one can understand the appeal. They can spend large sums quickly for relatively low “overhead” costs, and with increasing aid budgets and fewer staff to manage these, from a rather crude aid administration point of view, the attraction to challenge funds is perhaps explainable. But where is the impact and performance evidence to support this explosion of challenge fund programmes? I can’t find it. I’d like to suggest why I can’t:
- Input additionality: development problems are identified, and challenge funds put to task in response. They demand a level (typically 50%) of co-investment from grantees. They demand that the application demonstrates an innovation of some form, which the grantee can / is only prepared to invest in if public finance buys down some amount of real / perceived risk. Challenge funds are sold on their relevance to the identified development problem (i.e. funds are being put to an area recognised as problematic) and the fact that they are ‘leveraging’ partner investments (i.e. that they deliver input additionality). Yet, what does this really mean in practice when one considers a) a tiny proportion of partner investment is actually ‘investment’ in the real sense of the word: most of it is in-kind, working capital, or subordinated; and b) most grantees that I’ve ever interviewed are either institutionalised grantees, or aspire to be, having now found their way to the donor gravy train.
- Output additionality: incentives for robust oversight of challenge fund results are perverse. Anticipated challenge fund impacts are presented in proposals from grantees and decisions on grant awards are made. Awards are then communicated through a very visible and very public media ‘splash’ involving senior people from industry, donors and government. This publicity is a central and key feature of challenge funds – they are a competition after all – and is meant to uphold transparency. But, in my opinion, this very public process of awarding challenge funds means impact promises are ‘sold’ up front; and those implicated in selling this have little incentive to come clean and backtrack when reality doesn’t quite match anticipation. The fund manager doesn’t want to admit its due diligence was weak. The donor programme manager, who’s sold anticipated results up the political chain, doesn’t want to admit later that these estimations were perhaps overzealous. Comparable to wider experience there are no independent impact assessments of any DFID challenge funds (at least to my knowledge and in respect of those funds in the area of innovation or private sector development). Reported outputs remain anticipated, or those that are realised during the grant process tend to be because the grant itself is used as a direct subsidy (e.g. for free agricultural inputs, which are planted and grow more than last year’s seeds which had to be paid for). There’s little if any assessment of sustainability beyond the grant – probably because there wouldn’t be much to find, beyond perhaps the one or two exceptions used gratuitously to disprove the rule..!
- Behavioural additionality: funding from donors is not the same as funding from national governments. The UK Government might be willing to take a recurrent public funding role for innovation by SMEs in the UK. However, DFID funding in partner countries is transitory and not a public good in the same sense as UK Government funding made in the UK. The onus, then, must be on ensuring that behaviour change is realised, that it’s sustained. That ‘demonstration’ effects must be real and ‘replication’ must actually happen. Yet, it’s not clear that such change is at all considered in most challenge funds, let alone articulated, operationalised and measured, in spite of claims from many challenge funds that they now aim to be systemic and transformational.
Implications for DFID
Given the significant resources DFID is now pouring into challenge funds, and given the context of DFID demanding more accountability from the things it funds, more has to be done, and done quickly, to move beyond the rhetoric and into the reality of challenge funds. It’s not good enough to simply rely on relevance and input additionality claims. Evaluation and evidence must focus on the areas of output and behavioural additionality, and findings must feedback into modified designs. Specifically, I’d like to see a comparative evaluation of DFID challenge funds that focuses on:
- Rationale: confirming that a coherent economic rationale (rather than simply relevance) exists across its many and varied challenge funds.
- Results (output additionality): what these actually are, both during the grant (particularly when the grant is used as a direct transactional subsidy), and beyond the grant.
- Recurrence (behavioural additionality): were these grants actually transformational in any way, if not, could they be, if used more programmatically, or are they simply a convenient source of one-off grant funding to those well versed in accessing grant funding.
Why does such logic matter? Why is it not okay to simply continue to spend ever more grants in support of ever more disparate firms, to upgrade their production, and perhaps to boost their output a little? Well, I’ll leave you with the reflections of Adrian Wood, DFID’s former Chief Economist who commented in the year DFID prepared to launch its first private sector challenge fund in 1999: “My concern is simply that the phrase ‘assisting firm upgrading’ could be misconstrued by governments, donors and firms as an exhortation, for example, to spend public money on modern equipment for particular firms or to restrict competition. It needs to be made clear that this is not the intention”[2]. I wonder what Adrian would say about the rise and rise of challenge funds if he were still at DFID…?
[1] Written by Cunningham, Gok and Laredo, Manchester Institute of Innovation Research, August 2012
[2] Adrian Wood, “Value chains: an economist’s perspective”, DFID / IDS, 1999