Who Are We Excluding in Our Systems Designs

Systems Design has long been a fascination of mine. Whether that is technical, sociological, or philosophical—how do we codify that which we value and want to streamline and how does that impact those who don’t neatly fit into that mould. As I started my technical career building out website UIs, the topic of universal design captivated me. How are accessibility measures for some, appreciated by the many? Clear curb cuts serve those who rely on wheeled transport—wheelchairs, medical scooters, and more—but they also help parents with strollers, someone pulling a wagon full of supplies, and me pushing one of those small carts to get groceries.

For context, I don’t own a car and haven’t for the past two years. I have lots of thoughts about personal vehicles and urban design, but that’s a different topic entirely. I’m an athlete and do most of my local transportation by bike or public transit year round. Back in March, I did the classic thing of hitting some street car tracks wrong and finding myself heading over the handlebars. I was scraped up, a little sore, but I’d be fine. I continued with life, making small accommodations for myself and doing informal PT to help my sore body. I routinely push myself hard so my body being in pain isn’t a foreign concept. It takes a certain kind of masochism to willingly ride over 300km in a single day. I ended up making a casual comment in a meeting that I had never broken a bone. The irony ensues.

A week and a half after the accident, my thumb is still having a lot of painful flare ups. I’m supposed to do my first race of the season in the upcoming weekend. I reach out to my PCP thinking I need a referral to some more structured PT. I get set up with an in person appointment the next day for a quick exam, and am told I need to get an x-ray. I talk with my coach and a dear friend on the train ride home, debating if I should race or not. Ultimately, I opt out given the unknowns and not wanting to potentially jeopardize the safety of my fellow athletes. I sob on the train ride home, having one of my greatest joys and mental health management tools unexpectedly ripped away.

Disability meets Rigid Systems

Alright, so how does all this play into systems designs? 7 days after that doctor’s appointment I found myself receiving a phone call to come prepared the next day to go immediately from an urgent hand surgery consult to the hospital if I agreed to proceed with the surgery. I had about 20 hours of notice to remove piercings, notify my team, figure out a care plan, and a lot more. Thankfully, I had a wonderful support network which showed up for me. The surgical team had to get creative with their approach, but signs from the initial healing process almost 2 months post-op are looking promising. What I didn’t expect was the challenges of flying to a work on-site 10 days post-op.

Flying while trans is already an anxiety filled experience. I’m fortunate enough to afford TSA Pre-check to have the experience be slightly more humanizing. The anxiety is an ever-present droning noise in the background from stepping into the airport until I arrive in my hotel room. I’m used to that—as awful as such a statement is. This time was different, because I found myself traveling with a temporary physical disability. My AuDHD also creates a lot of travel based anxiety; while this disability is unseen it still comes into play. I do my cope ahead plans of how I’ll get to the gate early and ask for assistance to get my rolling bag into the overhead bins. I physically can’t, or really shouldn’t try to do that. I’m at my most susceptible for re-injury. The person I start talking to informs me that they can offer to check my bag free of charge. Checked luggage is a larger source of anxiety for me and isn’t something that my nervous system could handle. I politely decline and ask what other options are available.

It’s at this point that I found at, at least for this airline, that flight staff aren’t able to help put bags in the bins. Their shift starts when the cabin doors close, and this impacts potential worker’s compensation claims. I swear I have seen people helping others in the past, but arguing isn’t going to get me anywhere nor worth the limited energy reserves. The person reaffirms their offer to check my bag. I say that I will just have to hope a fellow passenger has good will to help me with my bag. I’m rapidly approaching meltdown levels, but just hold it together. Breathe. I just need to get to my seat, get my bag stored, and then I can put on my big headphones and re-regulate. Don’t have another Autistic meltdown in the airport or on the plane. Those are a miserable experience. A few minutes later, the pressure relief valve is unexpectedly released and I can breathe.

I see the person in front of me carrying a bag with a familiar company logo. I think I know who it is, but am not about to call out the wrong name. A bit later, I confirmed that it is a friend I’ve known for years. We managed to somehow find ourselves: on the exact same flight, to the same work event, and seated in the same row. She quickly offered to help me with my bag and we chatted on the entire 2 hour flight. I could breathe again.

Unconscious Bias Woven into the System

In my work, I use the coded gaze term as a reminder that the machines we built reflect the priorities, preferences, and even prejudices of those who have the power to shape technology. The coded gaze does not have to be explicit to do the job of oppression, including patriarchy and white supremacy, it is programmed into the fabric of society. Without intervention, those who have held power in the past continue to pass that power to those who are most like them. This does not have to be intentional to have a negative impact.

Dr. Joy Boulamwini, Unmasking AI

The concept of structural systems of exclusion and oppression aren’t anything new:

• Criminal Justice Reform continues to be an uphill battle.
• There are numerous studies and and statistics on how racial and ethnic minorities are disproportionately impacted by the criminal legal system.
• Intersectional Feminism has been a recurring source of concern raised by Audre Lorde in the 1970-1980s and currently with trans-feminism.
• The current hyperfixation of the American GOP party to strip transgender Americans of healthcare, including adults.
• Companies fighting to keep their apps inaccessible to blind and visually impaired individuals
• And so much more.

I’m only 70 pages into Dr. Joy Boulamwini’s amazing book. It highlights how the seemingly innocuous problems in current systems can have far greater concerns, including matters of life and death. Machine learning models which determine the existence or absence of a face may not seem too impactful. However, it points to a larger problem under the surface. Our systems are built by those in power, and inherently seek to maintain the status quo of their retention of power. Human life and experiences are reduced to a dollar figure. An acceptable form of collateral damage. Boulamwini identifies what happens for cases where the stakes are much higher. What happens when a machine learning model which seeks to identify civilians from militants is wrong? How does the increased usage of tools like this and others decenter the human costs of war and violence? The United Nations recently passed a resolution against autonomous weapons systems. The resolution raised concerns such as “negative consequences and impact of autonomous weapon systems on global security and regional and international stability, including the risk of an emerging arms race, of exacerbating existing conflicts and humanitarian crises, miscalculations, lowering the threshold for and escalation of conflicts and proliferation, including to unauthorised recipients and non-State actors” amidst other things.

Those are all Big Problems, Where do I have Agency?

Anecdotally, as someone who lives within systems which seek to marginalize me across multiple axes, a common question is what agency do I have about things? I’m not in the room where decisions are being made. My representatives aren’t ones who are pushing for stripping away my rights and humanity as a trans and queer woman. I live in a progressive state—I’m going to leave that commentary to the side. What can I do? I can still call my representatives and inform them of how current actions limit my travel and impact me. This is important work. I’d also argue that the foundation of systems can be approached from a more granular level.

As an engineer, and one with an interest in systems designs and iterations, a common interaction is the gathering of requirements, identifying edge cases, and designing a technical solution. A quick disclaimer: these are consistent cycles in the engineering process across different organizations, personal community engagements, and discussions within the larger tech community. My perspective is that these moments are value neutral. The purpose of this discussion is how do we engage in the process and consider diverse perspectives.

Systems constraints are a necessary mechanism for controlling flow logic, identifying boundaries, and determining which cases pass validation criteria. This can be as simple as verifying age for a service which is age restricted, serving up restaurants which are close to your current location for food delivery, etc. As committed engineers, we live and breathe these questions every single day. How should we handle these different cases? How does this feature come into the bigger picture direction of where we are heading? Is this a place to add technical debt for quick delivery? Have we already added substantial technical debt and this is a tipping point to prioritize consolidating things into a more sustainable solution? The best teams engage in this dynamic tension through cross disciplinary discussions. We likely have a shared goal of a great experience. If the balance becomes unmanageable, it can be challenging to recover. I’ve been fortunate to work with many highly communicative and collaborative teams.

How do we insert ourselves into the conversation? Some of my community work has been focused on discussing personal informational and operational security implications with a variety of people. Surveillance Capitalism has a lot of cascading and terrifying impacts. There are so many ways that the scary sides of our society’s technical architecture can be used against trans people. Texas has collected a database of trans individuals. How much privacy do services like Google, a staple of the online document management community, provide? Spoiler alert, it’s not great. Do we really have any alternatives? What knowledge blockers exist to prevent access or even knowledge of those alternatives to people? How can those barriers be reduced or eliminated? Related, I have some infosec and opsec lessons learned coming in future posts. If you are interested in doing a nuanced deep dive into the topic, I can highly recommend Michael Bazzell’s book: Extreme Privacy: What it Takes to Disappear. Yes it is an expensive book, but he keeps it updated almost annually which has a substantial cost. Please support his work directly if you can. I have no professional or paid affiliation.

Getting back to unconscious biases, we all have them. If we embrace diverse voices in different stages of system creation—technical, political, sociological—we enable the discussion of who our systems unintentionally, or sometimes intentionally, exclude. This doesn’t indicate explicit malicious intent, though that can be the case in the current political landscape against trans folx. By having diverse voices we can engage in mutual education, distribution of emotional labor, and increasing the quality of our services for a broader community. This is not the default. In the vein of universal design, as we seek to solve problems for certain people groups we often help many others. Systems Design is just as much a human problem as it is a technical problem. It may even be more of a human problem, asking the hard questions and engaging in the ethical questions of larger structural systems.

Default settings are not neutral. They often reflect the coded gaze the preferences of those who have the power to choose what subjects to focus on. But history has also shown us that alternative systems can be made. In the digital era, the LDK camera series developed by Phillips explicitly handled skin tone variation with two chips one for processing darker tones and another for processing lighter tones. The Oprah Winfrey Show used the LDK series for filming because there was an awareness of the need to better expose darker skin, given the show’s host and guests.

~ Dr. Joy Boulamwini, Unmasking AI