Speak Softly and Carry a Long Roadmap

"We want to be really innovative with data. The Mayor is serious about pedestrian safety, so I want to put city traffic incident data online and let people analyze it and build apps."

It was 2014. Bill de Blasio was a few months into his first mayoral term, and an acquaintance of mine had just become the deputy director of a major NYC agency. He invited me to his office to discuss his ideas, and this is one that he was especially passionate about.

He had also brought me to his office in order to recruit me. I was on my way out of CFPB, where our team was responsible for similar "big data" projects---that's what we called data projects back in 2014---like the Home Mortgage Disclosure Act. For all of these projects, the long pole in the tent was the back end: where does this data sit, how scalable is that architecture, and how big is the pipe going into it? If we didn't have good answers to those questions, the project was off to a bad start.

So I asked my friend where the traffic incident data was hosted.

"It's downstairs in a maintenance closet." Bad start!

Another guest spoke up with a more basic concern: "What problem are you trying to solve?"

His answer was meandering, but I'll always remember the words he said with the most conviction: "We need to be innovative."

When he said that, I knew the project had no chance of success. When someone tells me that their goal is to "be innovative," I'm immediately suspicious that their actual goal is to appear innovative to others---that they have a hammer and are in search of a nail, excited to try something new for its own sake, rather than trying to trying to solve a problem of public import. When a government technology project starts with this motivation, you often end up with a product that earns great headlines but doesn't help many people.

I have zero interest in sacrificing sustained success for fleeting feels. So instead of joining this man's agency, I joined a defense startup and somewhat lost touch with the civic innovation world for a few years.

I recently rejoined the civil service and looked around to see what had become of the landscape during my absence.

My reaction is mixed. Fifteen or so years ago, we set out to make government a viable career path for people who otherwise would have wound up at Instagram or Razorfish. We succeeded: within a niche community, we actually made government cool.

But instead of making government cool like NASA, we have made it cool like a party. Immensely talented digital professionals continue to sign up for public service, and there are numerous Federal offices built specifically to attract them: Presidential Innovation Fellows. U.S. Digital Service. 18F. AFWERX. Defense Innovation Unit.

What do all of these places do? Instead of assigning their immensely talented staff to a 10-year lunar mission, these organizations focus primarily on the near term. The civilian organizations tend to focus on single, high-visibility digital products---which is great, but given that these organizations are sponsored by offices with great authority over the Federal bureaucracy, I wish they would focus on the systemic problems that make digital production so difficult in the first place.

The military organizations are up to something different, and are faced with a somewhat different problem to solve. I'll try to cover this in a separate piece at a later date.

Civilian organizations

The general premise of USDS, 18F, and PIF is: if you're an agency and you need help with a digital product, call us. Our people can help you for a few months. They do other stuff, and some of it is along the lines of what I'm arguing for more of. But "fly-in" work is their focus.

This is a kind offer, but this is seldom the kind of help that agencies need for their digital projects. These groups swoop in and provide some mix of execution and ideas. But a few months? In Federal IT time, this is not worth much. And a newcomer providing ideas to experts is usually more of a distraction than a service; more on this later.

But let's assume that the average digital product ends up better as a result of these groups' contributions. Think about the opportunity cost. Are these piecemeal products really the best use of their time and power? The organizations behind these new digital teams carry enormous heft: GSA. OMB. The White House. The entire reason agencies need help on individual products is because their IT foundations are poor following decades of unpredictable budgets, slow hiring, and frustrating acquisitions. These are the hard problems that only GSA and OMB can fix. Turning it around takes years of consistent effort, which takes allies and political top cover---exactly what these new organizations can provide. I'm all for shipping product. But shipping great products would be far easier if we focused on the engines that power production. Stop helping agencies chop wood, and bring them some darn chainsaws, please.

Within these organizations, I'm sure it's invigorating to perpetually be in a start-up culture: there's always some hot new initiative that you know is going to get a lot of eyes. Everyone is brimming with ideas and the possibilities feel unlimited. And you're always in a race against time to ship product. Unfortunately, to those at the agency level who have been neck-deep in the policy domain and the IT architecture for years, the ideas of someone who just showed up are rarely valuable. This business model---"here are some more people, maybe they have some ideas to help you meet your deadline"---is precisely what software developers were taught not to do by one of the industry's classic texts, The Mythical Man-Month.

I've heard it argued that these fly-in teams exist partially in order to model agile development to agencies. This idea has some merit. An agency will only learn better ways of developing software if they see it done in their own organization. It's not enough to tell them about all the peer organizations that are doing things The Great New Way, because organizations are full of naysayers who like to say, "well it's nice that it worked at your agency, but it'll never happen here," almost as if they're proud of their intransigence.

So, if part of the mission of these organizations is to model good behavior, I'm all for it. But as long as this work is provided on a short-term basis, the benefits will be short-lived. Once an agile team departs, a stagnant organization will quickly revert to its decades-old practices without someone watching over them. To truly make progress in Federal IT, you need to think in years or even decades.

My message is not that short-term efforts have never provided value. But the balance is off. Instead of such a strong focus on getting individual products out the door, 18F and USDS should focus more on the foundational issues that keep Federal IT perpetually outmoded. When CFPB hired 30+ designers and developers in 2013, we envisioned two-year tenures for them. In my opinion, those two years were quite successful. But a selection of those 30+ fellows stuck around after their two years were up (after I had already left), and the work they've done since then has been far more impactful: instead of products, they have built enduring programs. Instead of designing single-use graphics, they have designed scalable processes and automated infrastructure. These achievements are the enablers of what we all want: reliably excellent digital products that help Americans get more out of their government. Without those enablers, shipping product is always going to feel perilous.

I recognize that the preference for short-term, high-visibility projects is often due to political forces outside our control. To anyone who is reading this who does have control over those forces: I know it's tempting to seize those big headlines at the cost of infrastructure that is largely invisible to the public and will deliver no tangible public benefit during your tenure: I have big things to achieve! Let my successors worry about those boring under-the-hood problems.

The trouble is that those successors never arrive. Your own predecessors sacrificed system stability for the sake of short-term victories, so that they could put those high-profile achievements on their résumés, thinking that their successors---you---would clean up their rushed architecture. Yesterday's decisions are making it hard for you to execute your own goals today.  Each time this happens, the technical debt grows. Who will have the guts to finally do this dirty work once and for all, while the nation's civic infrastructure still has a semblance of stability?

What to do instead

Right now, these innovation umbrella groups act as digital advisory firms to technically unsophisticated agencies, helping them build products on a piecemeal basis. Instead, they should be building culture and machinery: turn agencies into digitally savvy institutions that reliably ship excellent products without outside handholding. They can do this by focusing on three categories that leverage GSA's and OMB's strengths: recruiting people, providing standard platforms, and building physical spaces for digital product development.

People. Agencies need people who are in it for the long haul, who will take the time to learn the ins and outs of the architecture they're building on and the policy missions they're supporting. But every time an agency IT team attempts to hire great people, they end up reliving the same story: the applicants weren't good, or the application window wasn't open long enough, or the good candidates didn't make it past the first cut because the non-technical reviewer didn't know how to read a developer's résumé. So they cancel the posting and try it again.

The US Government needs a single recruiting team for digital: a single marketing budget, recruiters who know how to talk to the community, a single point of entry, and a delegated examining unit staffed with digital-savvy application evaluators. USDS's SME-QA project recognizes one aspect of the challenge, but it doesn't seem scalable. I'd rather have OPM post one cloud engineer position with hundreds of vacancies and share the cert USG-wide. I know USDS did a project with OPM in the past year or two, and that OPM recently created an office with a digital recruiting mission in mind. I hope this new outfit focuses on recruiting and hiring digital talent at scale, for the purpose of long-term placement within agencies. This is the long pole in the government IT tent, and it is a job that very few agency HR shops are succeeding at on their own. I'm optimistic about the U.S. Digital Corps, the newest USG-wide digital shop. It seems to have a model that resembles what I'm describing here.

Platforms. Many agencies are in a chicken-and-egg situation with respect to talent and infrastructure: they don't have the staff on hand to figure out what modern infrastructure looks like, but if they hired such staff to buy/build it, they would spend their first two years twiddling their thumbs waiting for the acquisition bureaucracy to deliver modern infrastructure with a developer sandbox.

Why, in the age of infrastructure-as-code and cloud.gov, does any federal agency lack a development environment? And not just a development environment, but a replica of its entire enterprise, including its unique mission systems? This is the type of environment that premier technology firms currently use to Red Team their stack. We should be using it as well, as a sandbox for every change, from device-level policy deviations to new development projects, with all of it tracked in source control. That should be the sniff test for any federal enterprise: how quickly can you build, deploy, and destroy a clone of your enterprise? If such a capability is not of interest to you, you've never had to resurrect your enterprise following a major breach.

Such a sandbox should especially be of interest to agencies that rely heavily on vendors, who often work on private networks where government customers have poor visibility of taxpayer-funded efforts. Think of it: instead of vague contractual requirements about which languages and platforms and security controls are to be used by prospective performers, just tell them: you shall build in our space, where everything you will need is waiting for you. Instead of ceaseless nail-biting about integrating and securing systems that have been built in disparate environments, just build everything in the same environment, with full-stack integration testing upon each and every commit. Include the U.S. digital design toolkit in every sandbox, as well as a single, standard devops and performance testing toolkit, making interagency integration smoother and more secure. This would also make Federal engineers' skills more portable to other agencies.

Would this cost a lot? Yes. But agencies have been duplicating one another's efforts here for several years, going through the high-cost motions of research and acquisition but ending with up with many incompatible systems. Getting agencies out of their digital stovepipes and onto shared, interoperable platforms, where we can truly leverage the cloud's economies of scale, is exactly what I'd expect these USG-wide digital organizations to be doing. Get the servers out of the closets!

Places. In a world where we've recruited an army of engineers and acquired a universal sandbox environment, those engineers would not even be able to access the environment described above, because their developer-unfriendly GFE devices wouldn't be able to connect to it. When working on Federal networks, physical location can mean everything. It's also critical when a big team is working on a complex project under a tight deadline. The answer is a shared space that is designed for digital product development.

Post-Covid, has the corporate world completely dismissed the virtues of colocated teams? I've been working remotely for almost 10 years. Most of that time has been spent with a private firm that develops remote collaboration software. And yet, I'm a remote work pessimist.

I've learned that small remote teams can be very efficient if the team members have established trust through prior in-person collaboration. But when the team gets so big that the members no longer know each other well, or the team is spread across multiple organizations, trust begins to erode. The team becomes less aware of various performers' contributions, which eventually turns into suspicion: What are those people doing all day? Why are there so many of them? Are they even doing anything?

I witnessed this recently while supporting Federal Student Aid's debt relief application. This product was a high priority for the White House, so USDS wanted to stay abreast of our work. Totally reasonable.

But imagine being USDS in this scenario: you show up at an organization with an architecture you've never worked on and with people you've never met. You immediately begin wondering: Who are these people? Are they good at what they do? When I ask them for updates, are they only telling me what I want to hear? Your job isn't to simply ask people how the work is going. Your job is to provide independent confirmation for the White House that the project is on track. But how do you do that when every interaction you have with the product team is virtual? The only thing you know is that someone in a meeting tells you that work is being done. But someone else's word isn't good enough for confirmation. You need to see the work for yourself.

Without a shared space in which to witness the work, how do you monitor it? Through meetings, mostly. And meetings aren't great for the productivity of the team (whose very productivity you are trying to ensure).

Even before Covid, when teams were clustered across a few sites, it was easy for factions to form and for drama to foment. In 2012, when CFPB overhauled its public web site, the team was spread across a few floors of the same building. To have an entire project team within one building, and all of them to be Feds from one agency, is unheard of. Yet even then, we were too spread out to work efficiently. As long as we were working from the offices of our various business units, we had distractions. The project was adrift until Peter Jackson, the assistant director for consumer engagement, used his own budget to rent the team a private space off-site. The dedicated space kept us on task and aware of our progress in every workstream. We stopped wasting time on status meetings. And because we all had visibility of one another's work, it did wonders for our trust in one another.

Every high-priority project in Federal IT needs its own space: a physical lab filled with the right hardware, the right network connectivity, enough for everyone on the team (including contractors), and VTC equipment to include those who absolutely must be remote. Make this space big enough to fit multiple digital product teams, and you suddenly have agencies watching, learning from, and feeding off of one another's breakthroughs. I can think of no better way to spread good digital practices across the government.

Can it be done? I hear GSA has some space available.