Wrote a SBIR blog. Then the DOD and members of Congress reached out, asked questions, and were all supportive. So let’s dig deeper and focus on the elephant in the room - SBIR mills.
You stated, "A niche market has developed for companies willing to pay to win SBIRs. Ex-DOD procurement officers and people that know the process act as third-party advisors to facilitate SBIRs — they fill out forms, make introductions, and find potential sponsors. The goal is to match-make with organizations, solicit Letters of Support (Phase I), and Memorandums of Understanding (Phase II)".
I wonder what your sources are for this assertion. It is not consistent with my experiences as SBIR PMs for many projects.
Also, you stated, "Most total awards for Phase I ( $50k) and Phase II ($750k) are the same across organizations." This statement is definitely incorrect.
For 1) I’m happy to email you 4-5 companies where this is exactly their business model, or call afwerx or navalx and ask. 2) yes, you are correct. army, mda, and navy can go higher, whereas >95% of AF is at 50k. Either way, it doesn’t change the outcomes of consolidation by a very small group of companies, especially in the navy. Thanks!
I would be interested to find out those companies. For many of the companies I have funded in the past, I was completely unaware of any of them who used this service. Perhaps, for my educational purpose, I would like to learn more and would appreciate your connection referrals to explore.
Ben, this is a pervasive practice, as noted here. However, it is most commonly applied to the "open topic" movement. That being said, I spent a small chunk of last calendar year and met with many SBIR-driven companies that work in the PNT space. They demonstrated no instincts, desire, will, or interest in pursuing the Defense Programs working on systems and platforms aligned with their topic. Therefore, the innovation they investigated - is not commercialized. It's a double-edged sword: so many of these entrepreneurs are talented academics (often start as faculty until their SBIR wins distract them), they are not "business" people. They'd thrive in a business as a technical leader. But, alas, they're an academic. They're doing the job of a lab researcher for solid profit and spawns a lifestyle and the only way to keep their lifestyle - and continue to reside in the bowels of their techincal discipline - is more grants with the labs. Who no doubt learn from them. I was skeptical at first of this phenomena - but after a round of many interviews in electrical engineering SBIR awardees - it became a pretty obvious trend. It makes sense if the grant enables the lab to learn and deepen its understanding of what can be done. If it's to commercialize innovation from the government spaces to the commercial marketplace to get back to the government - with the firms I've interviewed who work on special topics, not open topics - they're not doing that. Ironically - they need advisory and consulting services TO commercialize, not to WIN the grants on the topics they've mastered. Open-topics is a mix but opposite dynamic.
Oh this is *very* real. And why would a firm disclose this? In practice, even supposedly 'good' programs like AFWERX + DIU have senior folks revolve into private firms like secondfront, heavily developed by orgs like eaglepoint, and ghostwritten by various others.
We are R&D-heavy -- graph neural nets, GPU viz, etc used in enterprise+gov for core cyber/fraud/supplychain/etc data -- and realized the only viable way to engage with SBIR is paying the ~20% tax to these companies on top of the high amount of time. After seeing what does / doesn't make it through and some engagement attempts, we've increasingly viewed it as a waste of energy as an outsider due to all the effective nepotism going in. If the DC tax has to get paid either way, and SBIR odds are so bad as an outsider, might as well just work directly with primes who are already in the door and are at least honest about the cost of business.
Best to find customers, not distractors - bc the "DC tax," as you say, exists b/c it's discretionary funding. The president isn't asking about that spending - no one is - because it doesn't matter.
Thanks Amy! Excellent analysis and response. If we believe the new entrants argument regarding the challenges of working with the DOD (I do, I’ve felt it), and we look at the poor commercialization of the incumbents, we can and should also consider the inverse question: what can the DOD do to eliminate barriers of entry for new companies? From the first Phase 1 application, what if mechanisms were in place to begin deeper vetting and accreditation. Everyone talks about providing resources to span the valley of death, why not just shrink that valley ?
Funny you should mention barriers, I'm doing research on the barriers right now. My students used the SBIR web site information to identify companies that have completed either Phase I or II SBIRs between 2015 and 2022. We were able to send a survey and have ~280 responses. The survey questions are looking at two areas: barriers and transition characteristics. We are still neck deep in the analysis.
Beyond vetting, accreditation and barriers, I think that one underrated factor is having a plan. In a study of DIU data, one of my students (who was at DIU - great data access), found that not having a transition plan mapped directly to failed transition (https://scholar.afit.edu/etd/4345/). This evokes a drill sergeant barking out, 'if you fail to plan, you plan to fail.' DIU is focusing on fewer projects relative to SBIR, could we assume a magnification of the effect in the SBIR data set?
Next, towards the plan, as a professional hazard (I'm a Systems Engineer), I am a believer in open architectures and requirements; let's consider these integration plans. While not the solution for all Small Business efforts, I hypothesize that we could improve transition success if we think of the efforts as part of a whole and actually identify where the SBIR effort incorporates into the whole. A separate student research effort studied the effect of using system models in outreach to industry (Requests for Information). The firms were able to see where there product/tech fit into the larger picture. From the gov't side, the quality of information in the responses was higher and it was easier to identify which technologies were promising (https://scholar.afit.edu/etd/5551/).
I'll have to post more as we finalize our analysis on the barriers and transition characteristics.
Data point about transition from SBIRs to programs isn't bad on the company; it's 99% on the government's acquisition system.
Suggestion: don't limit P3's with your 8/4/2 theory; P3's are program funded (not SBA), can be procurement dollars, and aren't really part of the problem when looking at SBIR mills.
What's sad is that SBIR mills as intermediaries fill a market need because the government has made SBIRs unnecessarily complex.
Good discussion. You can actually find “SBIR Mills” referred to as “Frequent Winners” in the literature in 1990s. It was a more generous euphemism back then.
I have been doing research with SBIR data sets with students over the last few years. I can give you some numbers to go with your comment about commercialization for these investments (e.g. ’the success rate of these investments (Phase III and beyond) is hard to track but all indicators say it is abysmal’).
We did a review of 433 SBIR topics that completed Phase II between 2015 and 2018. The overall commercialization rate (I’m guessing that is what you mean by success) was 8.8%. Considering the diversity of technology, range of technical maturity and the fact that the DoD will play in monopsony markets, there is not a good measure of what the rate should be. However, the within cohort comparisons indicate that ‘frequent winning’ on the part of the contractors does not translate to success for the investments.
A common barrier cited by small businesses is the learning curve associated with the government; it’s kind of renowned or red tape and jargon. We decided to use number of government contracts awarded as a proxy for a company’s experience and ideally learning. We hypothesized that companies with few contracts would have a lower commercialization rate than those with more experience; this is and then isn’t the case.
We broke the data set into quartiles based on number of contracts and then computed the commercialization rates of the quartiles:
Q1 (1 to 4 contracts, n = 111): 7.2%
Q2 (5 to 14 contracts, n = 106): 13.2%
Q3 (15 to 35 contracts, n = 109): 7.3%
Q4 (36 to 419 contracts, n = 106): 7.5%
From Q1 to Q2 there is a significant jump, a near doubling, in commercialization. This seems to align with a learning curve hypothesis. However, the rates go back for the third and fourth quartiles. The ‘frequent winners’ or more specifically the companies with the most experience are statistically indistinguishable from the rookies. It drives policy questions regarding SBIR investments with these more experienced firms.
More details on the research will be in the upcoming issue of ARJ (Ryan, et al, 2022).
As a long time participant, with a very successful SBIR commercialization record, I think one of the biggest issues is that the SBIR program can be gamed by both the customer and the company to get around procurement hurdles. SBIRs are often just another way to buy something outside the normal funding or contracting processes (the O&M problem identified). One solution could be the addition of an "availability" hurdle - solicitations should allow a company to identify if they could / would deliver their proposed capability on a Firm Fixed Price basis (with a proposed price that could be contracted, to prevent gaming the other way). This would treat the SBIR topics like a sources sought, as well as an RFP, and identify topics which don't require innovation / development from SBIR funding. Once identified, those topics identified should be removed from the SBIR program, and either awarded or competed (at the Government's discretion) as a Small Business set-aside. The incentive for a company to identify a potential FFP effort would be the potential for a sole-source award if their price and proposal are appealing.
Have you thought about looking at this for agencies other than the DoD? Some of them have smaller budgets, but that makes these organizations monopolizing SBIR funds even more toxic for the development of new industry in the US. I'm on the gov side and honestly really dislike how the SBIR program is treated by some agencies. We'll have the people, facilities, and the idea to do something, and another center, after hearing of us, will basically write a call for a proposal from their favorite SBIR mill to recreate stuff we've already done and then abandon it at a Phase I/II. This is so much less efficient than funding the R&D directly at the government or just creating new vehicles for direct internal project to procurement at private industry.
I've also seen plenty of projects funded with no real final product in mind; the SBIR mill was just engaged to do a paper study or coarse model/prototype with no plan of it ever being fully procured. In this case, it's being used as a method to subsidize existing projects to do tasks they don't have the budget to fund themselves, under the guise of "small business development."
Yes, and I know the numbers are also terrible. Frankly, I just didn't have the time to look at all other agencies, as this is a random hobby. I also omitted STTRs from the DOD as well and the same companies violate things in the DOD.
Ahh, definitely fair enough! I've dug into some of these things on my side (being on the other side of the fence), and a lot of it is very depressing. The fundamentals of the program are such a great idea, but, like so many other projects dealing with the government, get gamed nonstop by grifters.
I'll say I 100% dead nuts caught a cause of fraud being attempted by a company, reported to the SBIR office, and the result was the firm just had to find a different subcontractor for that part of the project (about a $25k bid on something that should have cost $3k). Ignoring the fact the subcontract was to the brother of the president of the company, but said brother *had already done time in prison for fraud*. Was trying to get the company banned from bidding on SBIRs for at least a few years, but they said they don't like to do that sort of thing. =\
SBIRs seem like such a great idea... but is there something going on that we should know?
Maybe I am just disgruntled??? Maybe not.
I have submitted a number of SBIR proposals and have not yet :-) gotten one (though I once got very close!) and have observed some patterns. Now I do think that the people running the programs are trying to do the right thing but are they caught in a dilemma?
Case in point: I saw an SBIR topic from Air Force Research Lab (Kirtland) that paralleled some ideas that I had been proposing. So I put together a proposal and fired it off - but was not selected. The comments that I got were very odd, like they were comments about a different proposal. So I contacted AFRL and asked, they said that they were hosting a proposal from AFRL (Wright Pat) and could not answer any questions. Of course they volunteered to forward my questions to Wright Pat. I certainly think that my proposal could have solved the need that that they had and sure wish I could have talked to them about it.
But of course there is no way to have a dialog with the reviewers, no way to appeal their comments.
Part of the issue is that some of the evaluators have no business being evaluators, as their knowledge, skills, and abilities do not relate to the proposal at hand. The only requirement to evaluate a proposal is "to be a scientist or engineer." The system is broken in so many ways.
AFWERX is one of the worst offenders, IMO. The whole SBIR program seems like it acts as a backdoor way for the USG to contract with their preferred "small" vendors, while the intended purpose of seeding new innovations is left by the side of the road, chained to a tree.
Dan, everyone has their own experiences, but the data would signal that AFWERX is the best in terms of diversifying the portfolio of companies in the SBIR program. The Open Topic report I highlighted at the end of my article is worth a read. Hope things work out next time around.
Read your first article.
You stated, "A niche market has developed for companies willing to pay to win SBIRs. Ex-DOD procurement officers and people that know the process act as third-party advisors to facilitate SBIRs — they fill out forms, make introductions, and find potential sponsors. The goal is to match-make with organizations, solicit Letters of Support (Phase I), and Memorandums of Understanding (Phase II)".
I wonder what your sources are for this assertion. It is not consistent with my experiences as SBIR PMs for many projects.
Also, you stated, "Most total awards for Phase I ( $50k) and Phase II ($750k) are the same across organizations." This statement is definitely incorrect.
For 1) I’m happy to email you 4-5 companies where this is exactly their business model, or call afwerx or navalx and ask. 2) yes, you are correct. army, mda, and navy can go higher, whereas >95% of AF is at 50k. Either way, it doesn’t change the outcomes of consolidation by a very small group of companies, especially in the navy. Thanks!
Ben,
I would be interested to find out those companies. For many of the companies I have funded in the past, I was completely unaware of any of them who used this service. Perhaps, for my educational purpose, I would like to learn more and would appreciate your connection referrals to explore.
Thank you.
All it takes is a Google Search - https://www.sbiradvisors.com/ - this is hilariously sad.
Ben, this is a pervasive practice, as noted here. However, it is most commonly applied to the "open topic" movement. That being said, I spent a small chunk of last calendar year and met with many SBIR-driven companies that work in the PNT space. They demonstrated no instincts, desire, will, or interest in pursuing the Defense Programs working on systems and platforms aligned with their topic. Therefore, the innovation they investigated - is not commercialized. It's a double-edged sword: so many of these entrepreneurs are talented academics (often start as faculty until their SBIR wins distract them), they are not "business" people. They'd thrive in a business as a technical leader. But, alas, they're an academic. They're doing the job of a lab researcher for solid profit and spawns a lifestyle and the only way to keep their lifestyle - and continue to reside in the bowels of their techincal discipline - is more grants with the labs. Who no doubt learn from them. I was skeptical at first of this phenomena - but after a round of many interviews in electrical engineering SBIR awardees - it became a pretty obvious trend. It makes sense if the grant enables the lab to learn and deepen its understanding of what can be done. If it's to commercialize innovation from the government spaces to the commercial marketplace to get back to the government - with the firms I've interviewed who work on special topics, not open topics - they're not doing that. Ironically - they need advisory and consulting services TO commercialize, not to WIN the grants on the topics they've mastered. Open-topics is a mix but opposite dynamic.
Oh this is *very* real. And why would a firm disclose this? In practice, even supposedly 'good' programs like AFWERX + DIU have senior folks revolve into private firms like secondfront, heavily developed by orgs like eaglepoint, and ghostwritten by various others.
We are R&D-heavy -- graph neural nets, GPU viz, etc used in enterprise+gov for core cyber/fraud/supplychain/etc data -- and realized the only viable way to engage with SBIR is paying the ~20% tax to these companies on top of the high amount of time. After seeing what does / doesn't make it through and some engagement attempts, we've increasingly viewed it as a waste of energy as an outsider due to all the effective nepotism going in. If the DC tax has to get paid either way, and SBIR odds are so bad as an outsider, might as well just work directly with primes who are already in the door and are at least honest about the cost of business.
Best to find customers, not distractors - bc the "DC tax," as you say, exists b/c it's discretionary funding. The president isn't asking about that spending - no one is - because it doesn't matter.
Thanks Amy! Excellent analysis and response. If we believe the new entrants argument regarding the challenges of working with the DOD (I do, I’ve felt it), and we look at the poor commercialization of the incumbents, we can and should also consider the inverse question: what can the DOD do to eliminate barriers of entry for new companies? From the first Phase 1 application, what if mechanisms were in place to begin deeper vetting and accreditation. Everyone talks about providing resources to span the valley of death, why not just shrink that valley ?
Funny you should mention barriers, I'm doing research on the barriers right now. My students used the SBIR web site information to identify companies that have completed either Phase I or II SBIRs between 2015 and 2022. We were able to send a survey and have ~280 responses. The survey questions are looking at two areas: barriers and transition characteristics. We are still neck deep in the analysis.
Beyond vetting, accreditation and barriers, I think that one underrated factor is having a plan. In a study of DIU data, one of my students (who was at DIU - great data access), found that not having a transition plan mapped directly to failed transition (https://scholar.afit.edu/etd/4345/). This evokes a drill sergeant barking out, 'if you fail to plan, you plan to fail.' DIU is focusing on fewer projects relative to SBIR, could we assume a magnification of the effect in the SBIR data set?
Next, towards the plan, as a professional hazard (I'm a Systems Engineer), I am a believer in open architectures and requirements; let's consider these integration plans. While not the solution for all Small Business efforts, I hypothesize that we could improve transition success if we think of the efforts as part of a whole and actually identify where the SBIR effort incorporates into the whole. A separate student research effort studied the effect of using system models in outreach to industry (Requests for Information). The firms were able to see where there product/tech fit into the larger picture. From the gov't side, the quality of information in the responses was higher and it was easier to identify which technologies were promising (https://scholar.afit.edu/etd/5551/).
I'll have to post more as we finalize our analysis on the barriers and transition characteristics.
One small suggestion and one data point:
Data point about transition from SBIRs to programs isn't bad on the company; it's 99% on the government's acquisition system.
Suggestion: don't limit P3's with your 8/4/2 theory; P3's are program funded (not SBA), can be procurement dollars, and aren't really part of the problem when looking at SBIR mills.
What's sad is that SBIR mills as intermediaries fill a market need because the government has made SBIRs unnecessarily complex.
Good discussion. You can actually find “SBIR Mills” referred to as “Frequent Winners” in the literature in 1990s. It was a more generous euphemism back then.
I have been doing research with SBIR data sets with students over the last few years. I can give you some numbers to go with your comment about commercialization for these investments (e.g. ’the success rate of these investments (Phase III and beyond) is hard to track but all indicators say it is abysmal’).
We did a review of 433 SBIR topics that completed Phase II between 2015 and 2018. The overall commercialization rate (I’m guessing that is what you mean by success) was 8.8%. Considering the diversity of technology, range of technical maturity and the fact that the DoD will play in monopsony markets, there is not a good measure of what the rate should be. However, the within cohort comparisons indicate that ‘frequent winning’ on the part of the contractors does not translate to success for the investments.
A common barrier cited by small businesses is the learning curve associated with the government; it’s kind of renowned or red tape and jargon. We decided to use number of government contracts awarded as a proxy for a company’s experience and ideally learning. We hypothesized that companies with few contracts would have a lower commercialization rate than those with more experience; this is and then isn’t the case.
We broke the data set into quartiles based on number of contracts and then computed the commercialization rates of the quartiles:
Q1 (1 to 4 contracts, n = 111): 7.2%
Q2 (5 to 14 contracts, n = 106): 13.2%
Q3 (15 to 35 contracts, n = 109): 7.3%
Q4 (36 to 419 contracts, n = 106): 7.5%
From Q1 to Q2 there is a significant jump, a near doubling, in commercialization. This seems to align with a learning curve hypothesis. However, the rates go back for the third and fourth quartiles. The ‘frequent winners’ or more specifically the companies with the most experience are statistically indistinguishable from the rookies. It drives policy questions regarding SBIR investments with these more experienced firms.
More details on the research will be in the upcoming issue of ARJ (Ryan, et al, 2022).
As a long time participant, with a very successful SBIR commercialization record, I think one of the biggest issues is that the SBIR program can be gamed by both the customer and the company to get around procurement hurdles. SBIRs are often just another way to buy something outside the normal funding or contracting processes (the O&M problem identified). One solution could be the addition of an "availability" hurdle - solicitations should allow a company to identify if they could / would deliver their proposed capability on a Firm Fixed Price basis (with a proposed price that could be contracted, to prevent gaming the other way). This would treat the SBIR topics like a sources sought, as well as an RFP, and identify topics which don't require innovation / development from SBIR funding. Once identified, those topics identified should be removed from the SBIR program, and either awarded or competed (at the Government's discretion) as a Small Business set-aside. The incentive for a company to identify a potential FFP effort would be the potential for a sole-source award if their price and proposal are appealing.
Have you thought about looking at this for agencies other than the DoD? Some of them have smaller budgets, but that makes these organizations monopolizing SBIR funds even more toxic for the development of new industry in the US. I'm on the gov side and honestly really dislike how the SBIR program is treated by some agencies. We'll have the people, facilities, and the idea to do something, and another center, after hearing of us, will basically write a call for a proposal from their favorite SBIR mill to recreate stuff we've already done and then abandon it at a Phase I/II. This is so much less efficient than funding the R&D directly at the government or just creating new vehicles for direct internal project to procurement at private industry.
I've also seen plenty of projects funded with no real final product in mind; the SBIR mill was just engaged to do a paper study or coarse model/prototype with no plan of it ever being fully procured. In this case, it's being used as a method to subsidize existing projects to do tasks they don't have the budget to fund themselves, under the guise of "small business development."
Yes, and I know the numbers are also terrible. Frankly, I just didn't have the time to look at all other agencies, as this is a random hobby. I also omitted STTRs from the DOD as well and the same companies violate things in the DOD.
Ahh, definitely fair enough! I've dug into some of these things on my side (being on the other side of the fence), and a lot of it is very depressing. The fundamentals of the program are such a great idea, but, like so many other projects dealing with the government, get gamed nonstop by grifters.
I'll say I 100% dead nuts caught a cause of fraud being attempted by a company, reported to the SBIR office, and the result was the firm just had to find a different subcontractor for that part of the project (about a $25k bid on something that should have cost $3k). Ignoring the fact the subcontract was to the brother of the president of the company, but said brother *had already done time in prison for fraud*. Was trying to get the company banned from bidding on SBIRs for at least a few years, but they said they don't like to do that sort of thing. =\
Hit me up on LI to chat!
SBIRs seem like such a great idea... but is there something going on that we should know?
Maybe I am just disgruntled??? Maybe not.
I have submitted a number of SBIR proposals and have not yet :-) gotten one (though I once got very close!) and have observed some patterns. Now I do think that the people running the programs are trying to do the right thing but are they caught in a dilemma?
Case in point: I saw an SBIR topic from Air Force Research Lab (Kirtland) that paralleled some ideas that I had been proposing. So I put together a proposal and fired it off - but was not selected. The comments that I got were very odd, like they were comments about a different proposal. So I contacted AFRL and asked, they said that they were hosting a proposal from AFRL (Wright Pat) and could not answer any questions. Of course they volunteered to forward my questions to Wright Pat. I certainly think that my proposal could have solved the need that that they had and sure wish I could have talked to them about it.
But of course there is no way to have a dialog with the reviewers, no way to appeal their comments.
Part of the issue is that some of the evaluators have no business being evaluators, as their knowledge, skills, and abilities do not relate to the proposal at hand. The only requirement to evaluate a proposal is "to be a scientist or engineer." The system is broken in so many ways.
AFWERX is one of the worst offenders, IMO. The whole SBIR program seems like it acts as a backdoor way for the USG to contract with their preferred "small" vendors, while the intended purpose of seeding new innovations is left by the side of the road, chained to a tree.
Dan, everyone has their own experiences, but the data would signal that AFWERX is the best in terms of diversifying the portfolio of companies in the SBIR program. The Open Topic report I highlighted at the end of my article is worth a read. Hope things work out next time around.