HACKER Q&A
📣 fcc_throwaway

What to do about government agencies with terrible technical standards?


I currently work in a data engineering context that requires consuming data about FCC licenses from the FCC's ULS database. Understanding the data is crucial for complying with FCC licensing and transmission regulations. It appears to be a zipped collection of pipe-delimited files, but it cannot be parsed using any standard CSV parser, let alone specialized ones (like CleverCSV) which attempt to sniff out the correct format. (We've solved the problem with a custom parser).

Even if you succeed at parsing the files, the data quality is horrendous. Core data is missing. Geospatial records use non-standard serialization formats and often result in invalid polygons. Licensed frequencies may be outdated, or even impossible (such as lower_frequency > upper_frequency). There are cross-table inconsistencies. Incorrect codes are used everywhere.

Because the ULS doesn't actually process all of their rules regarding interference with adjacent licensees, sometimes a license that has perfect data still results in an incorrect representation of the license rights because an adjacent license has incorrect data. Therefore, the data quality problem is actually viral...bad data for one license can result in dozens of licenses not knowing their actual license rights.

This data is meant to be authoritative. It has been used in court cases and regulatory rulings. It is the first point of reference in license disputes. From our testing alone, which is not exhaustive, over 70% of licenses (in the subset of bands that we process) have at least one data quality error, and the virality of bad data effectively means nobody knows their license rights. What can actually be done about this? I'm starting to think that suing the FCC is the only way to get them to fix their problems, but that seems extreme. There have to be better options. Have any of you accomplished the task of getting the government to do a technical job better?

  👤 Jugurtha Accepted Answer ✓
Inform, guide, educate, and lobby relentlessly. Find allies and champions in the structures that are generating the data. Find a way to recruit them, onboard them, and make them care about this. Find a way to have them lobby their organization for better data formats, or anything that will make your job easier and make them successful.

Address the fear of being replaced by software and "AI" as soon as possible, and as frequently as possible. People view you and what you do as a threat to their position; convince them otherwise.

Frame problems in terms they understand. Find what they care about, and find how the work will impact what they care about.

This requires a systematic and consistent effort and positive energy. It is not easy to do the same pitch with the same enthusiasm for the sixth time on the same day at different places, but this is what it takes.

There's a level of trust and goodwill you have to build before anything gets done until it magically unlocks. You will doubt at times because the machinery moves very slow. You also need to be aware of how every person in every organization is thinking, what their motivations and incentives are, how the org chart works, how the "shadow org chart works", what they have to gain or lose, who's the decision maker (in title, in reality, not necessarily the same person) and who that decision maker listens to, who's competing for which position, who's in good or bad terms with who, who started this project and do they have support, did that person inherit the project and wants to "start over" with a clean slate so their predecessor does not get the credit. What position they're aiming for. This is how this works. This is not a CSV/JSON problem.