They Didn’t Stop To Think If They Should
Tim Gross | Machinist Labs
I’m Tim Gross. This is my twitter handle, which seemed like a good idea at the time. “They Didn’t Stop to Think If They Should”
or…
Machine Learning And The Internet Of Unpatched Things
or…
Because Eternal Vigilance is the Price of Liberty, We Have to Talk About Ethics Again
or maybe… “Because Eternal Vigilance is the Price of Liberty, We Have to Talk About Ethics Again”
We’ve updated our privacy policy!
Any reference to or citation of any person or organization does not constitute or imply an endorsement or recommendation of the content of this talk. The opinions expressed in this talk are the speaker’s alone and do not reflect the view of this conference, your employer, or my mom. The speaker is grossly unqualified to tell you how to live your life. Your mileage may vary. Not to be used in the manufacture of nuclear weapons. By attending this talk the speaker hereby grants you an irrevocable, perpetual, non-exclusive, transferable, worldwide license to be excellent to each other.
“Is this guy gonna talk about the trolley problem? C’mon, I just want to code! And who is this guy, anyways? He’s not an ethicist with a PhD!” Why do we as engineers need to discuss this stuff? And why is it especially important to talk about ethics when we start talking about ML and IoT? What makes these things special? What makes them different from other software? the reason we all need to be talking about this, is because as Sharon Kennedy Vickers told us yesterday: “what we do has impact”
Michael Laris, Washington Post, 24 May 2018
Back in March a self-driving car operated by Uber killed a pedestrian. The NTSB investigation is still ongoing, but what is clear is that the vehicle had no business being operated without direct supervision. The collision detection system had lots of false positives that caused it to brake erratically. So they turned the braking system off. But apparently that message never got to the folks who’d turned off the vehicles’ own automatic braking nor to the attendee of the vehicle. So the vehicle “saw” the pedestrian and did nothing.
James Vincent, The Verge, 24 Mar 2016
A couple years ago, MSFT demonstrated their ML capabilities with the Tay chatbot. They let it loose to learn from whatever racist trolls wanted to teach it. The experiment was stopped and the whole debacle embarrassed MSFT. Now clearly the horrible Internet people are primarily to blame here, but the researchers failed to anticipate the vulnerabilities inherent to their system
Russell Brandom, The Verge, 12 May 2017
in an IoT example, last year countless devices in UK hospitals were taken over by Wannacry ransomware. patient lives were put at risk not just because of poor patch hygiene but because of the architectural flaws and because of warped incentives in the US government program that produced the malware in the first place.
The Guardian, 31 Mar 2018
Tesla’s autopilot has been implicated in the deaths of several drivers at this point. We (and Tesla stockholders, apparently) are constantly reassured that these are the result of improper handling and not rushing these systems into the real world before they’re ready.
Kashmir Hill, Forbes, 16 Feb 2012
You didn’t think I was going to leave the home team out? Machine learning algorithms may know more about us and our loved ones than we do ourselves. Target was able to determine from customer purchases not just when customers are pregnant but at what stage of their pregnancy they were (buying unscented products during late stages, for example).
Jana Kasperkevic, The Guardian, 1 Jul 2015
ML is encoding our biases. Not just dehumanizing people online, but we saw yesterday how it can reinforce structures of power and privilege: access to mortgages, sentencing guidelines
what all these stories have in common is that like any other failure there’s almost certainly no “root cause”. these failures are the result of complex sociotechnical systems and the incentives they set up. it’s unlikely that anyone at these companies set out with awful intentions. But as Lanice Sims told us yesterday, your intentions ain’t shit. at the end of the day, it’s people in our industry – the people in this room – who are the ones who execute and implement these systems
a problem of scale
the problem isn’t just that these systems all have real-world consequences; software always has. we can go back to the 80’s and look at the Therac-25 accidents, where software flaws in radiation machines killed 3 people. the problem is that those consequences are multiplied by the scale of these systems.
quantity has its own quality
a huge part of the value proposition of IoT and ML is the scale of the data involved: collecting massive amounts of data from edge computing devices, and processing massive amounts of data in ML models.
but the scope of machine learning and IoT is incomprehensible to ordinary users. if you can determine through a ML model of someone’s purchases not just that they are pregnant but that they’re in the 3rd trimester, this isn’t a piece of data that the consumer willingly and knowingly shared with you, or have any way of knowing you could get. creating informed and meaningful consent is all but impossible
Robison Meyer, The Atlantic, 3 Oct 2013
because ML’s chaotic behavior is poorly understood, attacks on it can have open-ended results. maybe today someone is using a William Gibson’s Ugly T-Shirt to protect their identity from ubiquitous law enforcement use of facial recognition…
James Vincent, The Verge, 3 Jan 2018
what happens when a banana looks, not like a toaster, but a bomb or weapon? when an “accident” of that kind occurs, do the engineers of the system bear responsibility for failing to protect against this kind of “side channel” attack?
Kimberly Mok, The New Stack, 14 Sep 2017
the flaws of self-driving vehicles and the organizations operating them seem terrifying enough without adding adversarial environments into the mix. this is unremarkable graffiti…
The team found that with this approach, they were able to confuse a machine 100 percent of the time into classifying a stop sign as a 45-mile-per-hour speed limit sign, and a right-turn sign as a stop sign.
Kimberly Mok, The New Stack, 14 Sep 2017
but it was used to confuse a machine vision algorithm 100% of the time. these attacks are unreasonably effective!
embedded industry stuck in archaic threat model
embedded industry stuck in archaic threat model. they’re still shipping devices with shared private keys and hard-coded passwords. we used to say things like “well if you have physical possession then it’s game over” but that’s always the case with IoT devices. But we have answer to that: “secure boot” using TPM to sign the bootloader and OS updates. But this is treated as an expensive add-on rather than the default
http://www.eclipse.org/hawkbit/
existing solutions for IoT Over-the-Air updates (OTA) are mostly research projects at best (ex. Hawkbit, which depending on how you look at it is either an insecure-by-default toy, or an overcomplicated kit-of-parts)…
Our abdication of responsibility invites political remedy
the problem with our complacency on this as an industry is that it invites someone to “do something” about it.
“We have to do something!”
“This is something”
“We must do this!”
we’ve seen this over and over again. look at the evergreen fights over encryption, where the government thinks they can somehow make math available only to “Good People”
Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
https://www.apache.org/licenses/LICENSE-2.0
today much of our work is protected by saying “hey we don’t warranty this for fitness for any particular use.” (this is the Apache license but other OSS licenses are similar, as are most EULAs.) these agreements aren’t magic talismans; we’ve been allowed to get away with this, but that could change.
Do you want to be personally liable for bugs in your code?
https://news.ycombinator.com/item?id=16954306
top post on The Orange Site: “what many posters here miss is that a big group of tech people have no interest in dealing with legal matters.” aw, poor baby! you don’t get to be part of a world-impacting profession and pretend there are no real world consequences. childish!
feature, not bug
if you can’t do your job to protect the privacy of users and have to close up shop: Mission. Accomplished.
Salesforce CEO Marc Benioff thinks America needs “a national privacy law… that probably looks a lot like GDPR.
“This is going to help our industry… It’s going to set the guardrails around trust, around safety. It’s going to provide the ability for the customers to interact with great next generation technologies in a safe way.”
Simon Sharwood, The Register, 30 May 2018
It doesn’t have to be that way. In this interview with Marc Benioff he points out this can be good for our industry. It’ll “set the guardrails”
Benioff went on to say that as artificial intelligence is used in customer service, “that starts to cross the line on what is trust. And that’s where our industry really has to come forward and say we’re going to make sure that these technologies are trust-based. And I think the Europeans definitely got that figured out.”
Simon Sharwood, The Register, 30 May 2018
And this is deeply important because we as an industry have failed to set those guardrails for ourselves. And the larger society (our users) are starting to see this. It’s not too late. What can we do?
Consent
[10:00]
consent is the only workable guiding model when we’re talking about relationships between individual people. ex. I consent to being part of your research. You consent to engaging in conversation with me. When you decide you don’t want to be in a relationship you can withdraw consent and your partner respects that. We hopefully all understand this by now?
But consent has some limits once we get a lot more people involved. Although individual consent is the basis of liberal democracy, there are some times when we decide that the will of the community overrides the consent of an individual.
Steve Sack, Star Tribune, 27 Jan 2015
We expect everyone to pay their taxes. We ask that people are vaccinated. And the boundaries of individual vs community consent vary by culture. ex. in the EU they protect individual consent strongly…
Ilya Somin, Washington Post, 29 May 2015
whereas in the US we have a mixed bag where businesses (which are supposed to be individuals) are often given the backing of the community to override individual consent, without all the responsibility of consent.
https://www.google.com/search?q=externality
we see flaws of consent at play when we look at the Uber accident. the woman who was killed didn’t consent to be part of Uber’s experimental driving program. she wasn’t behind the wheel. what does “consent” mean when other human lives are treated as an externality? aside: what coal industry PR team infiltrated this into Google’s results?
foundational values?
the dynamic of community vs individual consent points to a problem defining foundational values (ref Bryan Cantrill’s Monktober talk distinguishes between “principles” and “values”). if we can disagree on things like the balance of individual vs community (ex. EU vs US), how do we define shared values? we’re not the first people to have this problem! so how do other professions solve this problem…?
“With great power comes great responsibility”
licensing as
self-regulation
historically licensing and professional organizations (ex. AMA, AIA, ASME, bar associations) have arisen from the professions themselves rather than being imposed clumsily from the outside. they’ve asked for the protection of regulation. government licensing requirements are typically delegated to the professional organizations
licensing requires monopoly
certifications do exist. we see lots and lots of useless rent-seeking certifications today already (CompTIA, Project Management Institute). but professional certification w/o the consequences of regulation is basically toothless. so before we get the government involved, it’s worth considering what the side-effects of regulation-supported monopoly would be.
We do not believe that it is merely a coincidence that the entry and standards of practice are most strictly regulated for physicians, dentists, and veterinarians… where the costs of receiving poor services could be high or sometimes even catastrophic.
ref http://www.nber.org/papers/w10467.pdf
study from the National Bureau of Economic Research shows that professional organizations have acted mostly as a counter to information asymmetry rather than creating monopoly power. consumers of our services generally don’t understand what they’re buying, and so professional bodies provide an answer to those questions. I’m not a doctor but I rely on the boards to tell me my doctor has met minimum standards of fair dealing, competence, and safety
gatekeeping
… so in other words they serve as gatekeepers. well, that’s a little problematic isn’t it? we have enough barriers to entry in our profession as it is. we still have a lot of work to do. so adding new barriers like “you must have this degree” or “you must have this many years experience working under a licensed developer” seem like they’d work against the direction we want to take our industry, right?
shared ethical baseline
but an advantage of a professional organization’s ethical framework is that it at least gives a shared baseline: right now we’re all trying to figure this out for ourselves. and we should hold each other to task
what is to be done?
[15:00]
waiting for regulation seems like a bad idea. relying on consent hasn’t really worked out. professional organizations seem fraught. but waiting for “the industry” to fix it isn’t working. we all, individually, are the industry. need to take individual action.
https://techworkerscoalition.org/
we can influence the community. there are activist organizations that have sprung up in the last few years looking to push for broader change across many organizations, like the Tech Workers Coalition
what about…?
“whataboutism”: because we have a restricted set of common values, it isn’t constructive to see e.g. Microsoftees protesting ICE but then turn around and say “what about their military contracts?” or “what about that time when MSFT’s CEO from 17 years ago said meanie-head things about open source?” This is not helpful. At least they’re doing something instead of snarking about it on twitter
Who’s hiring?
the labor market for our profession gives us enormous power right now. we can push hard for better hiring policies. we can push hard for D&I efforts. we can push hard for our organizations to be better
ensure your own mask is secure
that being said, we should also cut each other some slack. you don’t know much about the circumstances of any particular person (do they have health problems? family to take care of?). so while we should be holding each other to task, individuals need to make their own decisions about where they work and that doesn’t make them The Enemy. (unless they work at Palantir, just sayin’)
direct action
what is available to everyone regardless of our work conditions, and perhaps more effective than anything else we can do, is direct action at an engineering level. what choices do we make as technologists?
Immutable?
Jay Kreps, Cloudera, 11 June 2015
the pipelines we use to ingest data in many orgs are using immutable event stores. this means that once data is written it’s never really erased. if you’re keeping the entire commit history then you’re going to have a lot of problems with GDPR. (did you remember your backups?)
“Immutable”
but with changes at the application level, we can make this work. for example, we can use per-user encryption keys for a stream. this way we have encryption at rest, which is great, but we’ve also made our immutable event infrastructure compatible with the “right to be forgotten”: you delete the per-user key from the key store and the data is forever unreadable.
def should_brake(road):
if road.contains(object.HUMAN):
log.warn("oh shit!")
# TODO: this is causing erratic driving on false
# positive detection. Uncomment this once we have
# that solved. Someone should remind the field
# engineers to tell the test drivers they need to
# pay attention to the road.
# return True
return False
we need to take responsibility for quality proportional to the risks involved with the software. certainly this is obvious in directly-life-impacting software like self-driving cars, but it extends to every piece of software with real world impact. we need to be part of the chain of safety around these systems.
secure by design
we need to take responsibility for having deep understanding of how the security of our systems are bootstrapped. and we need to make decisions about that aligned with our values.
this means ensuring that your organization has PKI infrastructure in place to sign the bootloader and OS. getting familiar with secret stores like Hashicorp Vault to build the foundation of this infrastructure is a good practice
Jason Koebler, Motherboard, 21 Mar 2017
but there’s a tradeoff here. if you have secure boot that means your users will find it that much more difficult (if not impossible) to modify and repair the device firmware. this is a business decision and a decision about values; maybe the answer is different if the end user is a consumer vs a business? not making a decision one way or another means abdicating your responsibility as an engineer
secure communication
when we deploy IoT devices we need to ensure we’re taking advantage of modern TLS options. you wouldn’t transmit the login form for your web application over plain text, right? (right?!)
IoT devices typically have limited compute power, so folks avoid encryption. but we can solve this with modern choices: TLS1.3 reduces round-trips and elliptic curve keys use less memory than RSA keys. likewise, although you can cram MQTT into TLS, modern protocols like CoAP include mutually authenticated TLS. TLS isn’t “end-user facing” in IoT (unlike a browser), but secure communication isn’t about user optics but actually protecting the user.
def save(model, filename):
pickle.dump(model, open(filename, 'wb'))
and then when we’re done training, we take the result of all those weights and call it our model and just serialize the whole damn thing and ship it to prod. we can pass real world data in through the same model and out comes the classifications.
ML models are state
our entire industry has unified around our worries about statefulness. “run your applications as stateless containers! with k8s! let your cloud provider lock-in – I mean securely host – all your stateful applications!” But ML is the ultimate stateful application. you’re using software to generate these software models and you ship those opaque blobs. let’s get the Functional Programming crowd in a twist: the entire application is a side-effect! how can we influence those side-effects?
https://ai.google/research/pubs/pub43146
we desperately need better tooling to design-out unexpected behaviors in ML. We have undeclared consumers/dependencies and hidden “strange loops” of feedback. These represent a source of technical debt. and it’s the worst kind of technical debt – it’s what John Allspaw and the SNAFUcatchers call “shadow debt” that’s taken on unknowingly.
To make great products:
do machine learning like the great engineer you are, not like the great machine learning expert you aren’t.
…
Insofar as well-being and company health is concerned, human judgement is required to connect any machine learned objective to the nature of the product you are selling and your business plan.
Martin Zinkevich (Google), Rules of Machine Learning: Best Practices for ML Engineering
we need to remember with any ML project that human judgement is required to connect the algorithms to their impact with the real world – our business, our industry, and the community as a whole
SQL > ML
first rule should always be: why are we choosing ML over some well-tuned SQL or other algorithm? why are we choosing to use a method with chaotic feedback mechanisms instead of something that’s simple and deterministic? imagine explaining to your CTO why you chose a web framework that allowed for undefined behavior on hostile inputs! is this simply because of resume-driven development?
models should be testable and human-interpretable
when we do choose ML, we should choose our approach with interpretability in mind. simple linear or logarithmic regression models are easier to debug and avoid unexpected feedback loops than models that self-optimize. remove unused features. quantify any unexpected behavior and build tests for it. these behaviors are both technical debt and avenues for adversarial input
align training data with real world demographics
choose ML training inputs that reflect the population. this is a fortunate case where we can easily align engineering ethics with business needs in a way the business understands (ex. “if we pick machine vision training data that reflects real demographics, we can avoid the embarrassment for our organization of having to explain why our software acts racist.”) win-win, eh?
technical leadership
we can’t rely on project managers or business analysts to take the lead on designing our systems ethically because they simply may not understand the side-effects. you’re the technical professional: they’re expecting you to take the lead on this!
“best practice is…”
“the best thing about best practices is there are so many of them to choose from.” you don’t need to ask permission from your business analysts and project managers on opinions that are purely technical
We aren’t a craft anymore. We might feel like artisans with laptops but what we produce could potentially be in front of a significant chunk of the human race by lunchtime. We’re not hand-crafting dovetail joints here.
Anne Currie, The Register, 1 Mar 2018
leave you with this from Anne Currie at ContainerSolutions. “We aren’t a craft anymore. We might feel like artisans with laptops but what we produce could potentially be in front of a significant chunk of humanity by lunchtime.”
let’s get to work
what we do has impact. let’s get to work. thanks, folks!
