Wrote a SBIR blog. Then the DOD and members of Congress reached out, asked questions, and were all supportive. So let’s dig deeper and focus on the elephant in the room - SBIR mills.
You stated, "A niche market has developed for companies willing to pay to win SBIRs. Ex-DOD procurement officers and people that know the process act as third-party advisors to facilitate SBIRs — they fill out forms, make introductions, and find potential sponsors. The goal is to match-make with organizations, solicit Letters of Support (Phase I), and Memorandums of Understanding (Phase II)".
I wonder what your sources are for this assertion. It is not consistent with my experiences as SBIR PMs for many projects.
Also, you stated, "Most total awards for Phase I ( $50k) and Phase II ($750k) are the same across organizations." This statement is definitely incorrect.
Thanks Amy! Excellent analysis and response. If we believe the new entrants argument regarding the challenges of working with the DOD (I do, I’ve felt it), and we look at the poor commercialization of the incumbents, we can and should also consider the inverse question: what can the DOD do to eliminate barriers of entry for new companies? From the first Phase 1 application, what if mechanisms were in place to begin deeper vetting and accreditation. Everyone talks about providing resources to span the valley of death, why not just shrink that valley ?
Data point about transition from SBIRs to programs isn't bad on the company; it's 99% on the government's acquisition system.
Suggestion: don't limit P3's with your 8/4/2 theory; P3's are program funded (not SBA), can be procurement dollars, and aren't really part of the problem when looking at SBIR mills.
What's sad is that SBIR mills as intermediaries fill a market need because the government has made SBIRs unnecessarily complex.
Good discussion. You can actually find “SBIR Mills” referred to as “Frequent Winners” in the literature in 1990s. It was a more generous euphemism back then.
I have been doing research with SBIR data sets with students over the last few years. I can give you some numbers to go with your comment about commercialization for these investments (e.g. ’the success rate of these investments (Phase III and beyond) is hard to track but all indicators say it is abysmal’).
We did a review of 433 SBIR topics that completed Phase II between 2015 and 2018. The overall commercialization rate (I’m guessing that is what you mean by success) was 8.8%. Considering the diversity of technology, range of technical maturity and the fact that the DoD will play in monopsony markets, there is not a good measure of what the rate should be. However, the within cohort comparisons indicate that ‘frequent winning’ on the part of the contractors does not translate to success for the investments.
A common barrier cited by small businesses is the learning curve associated with the government; it’s kind of renowned or red tape and jargon. We decided to use number of government contracts awarded as a proxy for a company’s experience and ideally learning. We hypothesized that companies with few contracts would have a lower commercialization rate than those with more experience; this is and then isn’t the case.
We broke the data set into quartiles based on number of contracts and then computed the commercialization rates of the quartiles:
Q1 (1 to 4 contracts, n = 111): 7.2%
Q2 (5 to 14 contracts, n = 106): 13.2%
Q3 (15 to 35 contracts, n = 109): 7.3%
Q4 (36 to 419 contracts, n = 106): 7.5%
From Q1 to Q2 there is a significant jump, a near doubling, in commercialization. This seems to align with a learning curve hypothesis. However, the rates go back for the third and fourth quartiles. The ‘frequent winners’ or more specifically the companies with the most experience are statistically indistinguishable from the rookies. It drives policy questions regarding SBIR investments with these more experienced firms.
More details on the research will be in the upcoming issue of ARJ (Ryan, et al, 2022).
As a long time participant, with a very successful SBIR commercialization record, I think one of the biggest issues is that the SBIR program can be gamed by both the customer and the company to get around procurement hurdles. SBIRs are often just another way to buy something outside the normal funding or contracting processes (the O&M problem identified). One solution could be the addition of an "availability" hurdle - solicitations should allow a company to identify if they could / would deliver their proposed capability on a Firm Fixed Price basis (with a proposed price that could be contracted, to prevent gaming the other way). This would treat the SBIR topics like a sources sought, as well as an RFP, and identify topics which don't require innovation / development from SBIR funding. Once identified, those topics identified should be removed from the SBIR program, and either awarded or competed (at the Government's discretion) as a Small Business set-aside. The incentive for a company to identify a potential FFP effort would be the potential for a sole-source award if their price and proposal are appealing.
Have you thought about looking at this for agencies other than the DoD? Some of them have smaller budgets, but that makes these organizations monopolizing SBIR funds even more toxic for the development of new industry in the US. I'm on the gov side and honestly really dislike how the SBIR program is treated by some agencies. We'll have the people, facilities, and the idea to do something, and another center, after hearing of us, will basically write a call for a proposal from their favorite SBIR mill to recreate stuff we've already done and then abandon it at a Phase I/II. This is so much less efficient than funding the R&D directly at the government or just creating new vehicles for direct internal project to procurement at private industry.
I've also seen plenty of projects funded with no real final product in mind; the SBIR mill was just engaged to do a paper study or coarse model/prototype with no plan of it ever being fully procured. In this case, it's being used as a method to subsidize existing projects to do tasks they don't have the budget to fund themselves, under the guise of "small business development."
SBIRs seem like such a great idea... but is there something going on that we should know?
Maybe I am just disgruntled??? Maybe not.
I have submitted a number of SBIR proposals and have not yet :-) gotten one (though I once got very close!) and have observed some patterns. Now I do think that the people running the programs are trying to do the right thing but are they caught in a dilemma?
Case in point: I saw an SBIR topic from Air Force Research Lab (Kirtland) that paralleled some ideas that I had been proposing. So I put together a proposal and fired it off - but was not selected. The comments that I got were very odd, like they were comments about a different proposal. So I contacted AFRL and asked, they said that they were hosting a proposal from AFRL (Wright Pat) and could not answer any questions. Of course they volunteered to forward my questions to Wright Pat. I certainly think that my proposal could have solved the need that that they had and sure wish I could have talked to them about it.
But of course there is no way to have a dialog with the reviewers, no way to appeal their comments.
Read your first article.
You stated, "A niche market has developed for companies willing to pay to win SBIRs. Ex-DOD procurement officers and people that know the process act as third-party advisors to facilitate SBIRs — they fill out forms, make introductions, and find potential sponsors. The goal is to match-make with organizations, solicit Letters of Support (Phase I), and Memorandums of Understanding (Phase II)".
I wonder what your sources are for this assertion. It is not consistent with my experiences as SBIR PMs for many projects.
Also, you stated, "Most total awards for Phase I ( $50k) and Phase II ($750k) are the same across organizations." This statement is definitely incorrect.
Thanks Amy! Excellent analysis and response. If we believe the new entrants argument regarding the challenges of working with the DOD (I do, I’ve felt it), and we look at the poor commercialization of the incumbents, we can and should also consider the inverse question: what can the DOD do to eliminate barriers of entry for new companies? From the first Phase 1 application, what if mechanisms were in place to begin deeper vetting and accreditation. Everyone talks about providing resources to span the valley of death, why not just shrink that valley ?
One small suggestion and one data point:
Data point about transition from SBIRs to programs isn't bad on the company; it's 99% on the government's acquisition system.
Suggestion: don't limit P3's with your 8/4/2 theory; P3's are program funded (not SBA), can be procurement dollars, and aren't really part of the problem when looking at SBIR mills.
What's sad is that SBIR mills as intermediaries fill a market need because the government has made SBIRs unnecessarily complex.
Good discussion. You can actually find “SBIR Mills” referred to as “Frequent Winners” in the literature in 1990s. It was a more generous euphemism back then.
I have been doing research with SBIR data sets with students over the last few years. I can give you some numbers to go with your comment about commercialization for these investments (e.g. ’the success rate of these investments (Phase III and beyond) is hard to track but all indicators say it is abysmal’).
We did a review of 433 SBIR topics that completed Phase II between 2015 and 2018. The overall commercialization rate (I’m guessing that is what you mean by success) was 8.8%. Considering the diversity of technology, range of technical maturity and the fact that the DoD will play in monopsony markets, there is not a good measure of what the rate should be. However, the within cohort comparisons indicate that ‘frequent winning’ on the part of the contractors does not translate to success for the investments.
A common barrier cited by small businesses is the learning curve associated with the government; it’s kind of renowned or red tape and jargon. We decided to use number of government contracts awarded as a proxy for a company’s experience and ideally learning. We hypothesized that companies with few contracts would have a lower commercialization rate than those with more experience; this is and then isn’t the case.
We broke the data set into quartiles based on number of contracts and then computed the commercialization rates of the quartiles:
Q1 (1 to 4 contracts, n = 111): 7.2%
Q2 (5 to 14 contracts, n = 106): 13.2%
Q3 (15 to 35 contracts, n = 109): 7.3%
Q4 (36 to 419 contracts, n = 106): 7.5%
From Q1 to Q2 there is a significant jump, a near doubling, in commercialization. This seems to align with a learning curve hypothesis. However, the rates go back for the third and fourth quartiles. The ‘frequent winners’ or more specifically the companies with the most experience are statistically indistinguishable from the rookies. It drives policy questions regarding SBIR investments with these more experienced firms.
More details on the research will be in the upcoming issue of ARJ (Ryan, et al, 2022).
As a long time participant, with a very successful SBIR commercialization record, I think one of the biggest issues is that the SBIR program can be gamed by both the customer and the company to get around procurement hurdles. SBIRs are often just another way to buy something outside the normal funding or contracting processes (the O&M problem identified). One solution could be the addition of an "availability" hurdle - solicitations should allow a company to identify if they could / would deliver their proposed capability on a Firm Fixed Price basis (with a proposed price that could be contracted, to prevent gaming the other way). This would treat the SBIR topics like a sources sought, as well as an RFP, and identify topics which don't require innovation / development from SBIR funding. Once identified, those topics identified should be removed from the SBIR program, and either awarded or competed (at the Government's discretion) as a Small Business set-aside. The incentive for a company to identify a potential FFP effort would be the potential for a sole-source award if their price and proposal are appealing.
Have you thought about looking at this for agencies other than the DoD? Some of them have smaller budgets, but that makes these organizations monopolizing SBIR funds even more toxic for the development of new industry in the US. I'm on the gov side and honestly really dislike how the SBIR program is treated by some agencies. We'll have the people, facilities, and the idea to do something, and another center, after hearing of us, will basically write a call for a proposal from their favorite SBIR mill to recreate stuff we've already done and then abandon it at a Phase I/II. This is so much less efficient than funding the R&D directly at the government or just creating new vehicles for direct internal project to procurement at private industry.
I've also seen plenty of projects funded with no real final product in mind; the SBIR mill was just engaged to do a paper study or coarse model/prototype with no plan of it ever being fully procured. In this case, it's being used as a method to subsidize existing projects to do tasks they don't have the budget to fund themselves, under the guise of "small business development."
SBIRs seem like such a great idea... but is there something going on that we should know?
Maybe I am just disgruntled??? Maybe not.
I have submitted a number of SBIR proposals and have not yet :-) gotten one (though I once got very close!) and have observed some patterns. Now I do think that the people running the programs are trying to do the right thing but are they caught in a dilemma?
Case in point: I saw an SBIR topic from Air Force Research Lab (Kirtland) that paralleled some ideas that I had been proposing. So I put together a proposal and fired it off - but was not selected. The comments that I got were very odd, like they were comments about a different proposal. So I contacted AFRL and asked, they said that they were hosting a proposal from AFRL (Wright Pat) and could not answer any questions. Of course they volunteered to forward my questions to Wright Pat. I certainly think that my proposal could have solved the need that that they had and sure wish I could have talked to them about it.
But of course there is no way to have a dialog with the reviewers, no way to appeal their comments.