Falling Colors Technology a BHSD Crony that needs Competition

BHSD (Behavioral Health Services Division) is a New Mexico state agency that doles out federal and state funds to a variety of small, ostensibly health related, programs. For example, in the state of New Mexico, BHSD runs a program called Synar¹ that attempts to cut down on merchants selling cigarettes to minors. One Falling Colors employee characterized the program as “mostly stick and no carrots.” Synar funds a stable of ambush inspectors that descend on merchants hoping to catch them selling to minors. It’s a standard bit of well-intentioned government coercion. If you are wondering what’s in it for the merchants stop wondering: it’s all stick. They lose sales and face fines. If you are wondering why the state of New Mexico supports Synar follow the money. Depending on the dubious statistics compiled by Synar administrators the state could lose millions of dollars of federal grants if the percentage of offending merchants exceeds an arbitrary threshold. Synar, in the twisted minds of state bureaucrats, “generates revenue.”

In addition to saving the state from the unconscionable scourge of teenage smoking BHSD also funds a mishmash of programs to prevent drug addiction, help the mentally ill, subsidize methadone treatments and reimburse psychologists, psychiatrists and other health professionals for counseling and other services. BHSD’s budget for all these operations is, according to Mindy Hale, roughly fifty million dollars per year. In the greater wasteful schemes of government this is small beans, even for New Mexico, but, it’s still fifty million public dollars so it’s not out-of-bounds to ask, what are the taxpayers getting for their money?

If you are naïve enough to think the intended clients of BHSD’s largesse, the teenage smokers, the mentally ill, and the drug addicts, garner the lion’s share of that fifty million dollars you’re probably a statist or a moron, but I repeat myself. Many years ago a wise old wag, when badgered about the high cost of landing a man on the moon, chirped, “None of that money was spent on the moon!” While some of BHSD’s fifty million is directed to clients, the moon, the lion’s share goes to contractors, service providers, and BHSD internals. Whether the state of New Mexico and the federal government are getting good value for their money is debatable; what’s not debatable is that some IT service providers are doing very well from themselves.

Two IT providers consume a significant share of BHSD IT funds: Optum New Mexico and Falling Colors Technology. The founders of Falling Colors, Mindy Hale and Pamela Koster², claim Optum bills the state of New Mexico roughly four million dollars per year for the onerous job of cutting checks. It’s important to understand that Optum is not dispensing their own money. They are simply managing a pool of funds that are replenished by state and federal tax dollars. Yes, it takes money to manage money. You have to pay auditors, comptrollers, and other financial professionals to make sure the funds are not redirected into questionable pockets. Surely you don’t think New Mexico’s corruption free government would abscond with unwatched dollars?

Still, four million seems a bit steep for providing a routine service that any experienced financial entity like a bank could do, and to the state’s credit, they have recognized this and are in the process of renegotiating Optum’s four million dollar fee. Optum has responded with a “this isn’t worth the damn hassle” attitude. If they cannot get their four million they’re threatening to pull out of the state and cede the check cutting business to others. How much of this is hardball negotiating, corporate whining, or even the truth, is hard to determine. The only thing that seems certain is that there is a business opportunity for an IT provider if Optum makes good on their threat and leaves New Mexico.

Falling Colors Technology, a little company that is already extracting about one million dollars per year from the state, is angling to take over Optum’s fund disbursement role. In standard insider crony fashion, they hope to keep this transfer quiet and elude potential competitors. Why go through all that messy inefficient public bidding? There’s only one problem with their business plan. Falling Colors has absolutely no experience managing funds. There is nobody on their staff that could be considered a financial professional. They are planning to hire staff, but I have to wonder why BHSD, and the state of New Mexico, are considering flushing millions of dollars through an entity that has no financial expertise and has already received a formal letter of warning for shoddy IT work.

Instead of branching out into lines of business that they have no experience with Falling Colors efforts would be better invested in fixing their core problems and they have lots of core problems. Let’s look at what nearly one million dollars or public funds per year buys from Falling Colors Technology.

Your one million is buying a few unreliable, crash prone, insecure, low volume websites geared towards BHSD staff and service providers. When I first ran the following SQL query on the database that backs many Falling Colors websites I was alarmed at the results.

SELECT  iq.WeekNumber
        , AVG(iq.ErrorCount) AS AvgWeekErrors
        , MIN(iq.ErrorCount) AS MinWeekErrors
        , MAX(iq.ErrorCount) AS MaxWeekErrors
        , STDEV(iq.ErrorCount) AS StdDevWeekErrors
FROM (SELECT CAST(CONVERT(VARCHAR(8), TimeUtc, 112) AS INTEGER) 
           AS DayNumber 
         , COUNT(1) AS ErrorCount
         , MIN(DATEPART(iso_week, TimeUtc)) AS WeekNumber
      FROM  dbo.ELMAH_Error
      GROUP BY CAST(CONVERT(VARCHAR(8), 
                    TimeUtc, 112) AS INTEGER)
     ) iq
GROUP BY iq.WeekNumber

Falling Colors websites were crashing about twenty times per day. On some days the crash count exceeded fifty. I thought to myself, “If this doesn’t dramatically improve this little company is doomed.” I’ve worked with lots of bug infested software over my long career but twenty to fifty crashes per day, distributed over a few dozen users, was an entirely new level of unreliability.

Why is it so bad? The developers at Falling Colors, like developers everywhere, bitched about “inherited code.” Basically, this means they’re working with code that they didn’t entirely write themselves. Developers complaining about inherited code is so common that software managers rightly label it whining. Software developers bitching about inherited code is like civil engineers griping about inherited bridges. The world is not created fresh every day. The inherited code base is a source of problems but the main reason Falling Colors exhibits such a high crash rate is simply a lack of formal quality control.

Testing at Falling Colors is mostly performed by one beleaguered Business Analyst. She runs through a series of basic web page checks after significant new releases. This is a very low standard of testing for modern software development. Falling Colors does not practice many common quality control techniques. For example, most development environments support a variety of internal testing tools. Falling Colors is a Visual Studio shop and Visual Studio has built-in unit testing tools and supports a host of third-party add-ons. Developers focused on quality, spend as much time implementing internal units test as they do writing production code. There is an entire coding regime known as TDD that strongly promotes writing tests before you write software to pass the tests. At the end of June 2016, there were precisely zero internal unit tests in Falling Color’s code base. In addition to missing internal unit tests, there were no repeatable or scripted tests, no large case tests, and no stress tests. Lack of formal testing combined with misplaced developer optimism is a recipe for high error rates and Falling Colors is really boiling that pot.

Buggy insecure low volume websites are a dime a dozen. There’s a lot of crap out there. If Falling Colors cranked out standard public websites we would click on and ignore their rubbish. Unfortunately, being intertwined with BHSD, the users of Falling Colors websites do not have the option of clicking on. Making things worse, Falling Colors hosts a substantial amount of HIPAA protected information.

HIPAA is a set of federal guidelines that outline how health providers and their contractors must protect information that might be used to uniquely identify people. HIPAA penalties, for both providers and individuals, are severe if protected information is either accidentally or willfully disclosed. You can go to jail for exposing HIPAA protected information.

HIPAA guidelines list common data elements that must be protected. There is only one way to properly protect these elements: full element encryption. Every single data element should be encrypted and the keys should be rigorously guarded by a small number of individuals. Even developers, especially developers, should never see the unencrypted information. This is the way things should work, but, if you have followed the news about an unending stream of website hacks and data breaches, you’re probably aware that this is not how it works in the big nasty world.

It’s certainly not the way things are working at Falling Colors. With the exception of website passwords, which were only hashed in the last year,³ HIPAA data is stored in plain, ready to hack, text. If I were an IT savvy methadone user in the state of New Mexico I would be reluctant to disclose personal information to CareLink, TreatFirst, Prevention, or any of the Falling Colors managed programs. One HIPAA breach and your methadone habit is on Facebook.

Falling Colors is fully cognizant of their shabby security and are planning to eventually fix it. They’re taking steps to harden their websites and tighten up their loose databases but they are not, as of the end of June 2016, pursuing a full element encryption regime. Anything short of full element encryption is just putting lipstick on the security pig. Currently, Falling Colors is a HIPAA breach in waiting. BHSD would be well advised to insist on an immediate and independent full security audit of Falling Colors systems!

BHSD should also demand a fair and public RFP (Request for Proposal) process when seeking IT contracting services. Currently, some individuals in BHSD, in connivance with Falling Colors, are delicately crafting RFPs that are designed to exclude Falling Colors competitors. This is a blatant abuse of the public RFP process and the perpetrators should be ashamed of themselves. Crony state contracting may be business as usual in New Mexico but it is not in the interests of the pubic, BHSD, or even Falling Colors. Cronies without competition invariably turn into parasites and BHSD, which recently suffered a bedbug outbreak in their Santa Fe offices, has enough of those.