Organisationally it never works to have a group whose only job is to say no to some other group. The incentives are diametrically opposed and as a structure it can’t last.
If you had an AI company and want it to be ethical you have to find a way to make ethics everyone’s responsibility, and have the consequences of poor ethics bite the people who make those bad decisions. If you just outsource it to the ethics group what happens is
1)everyone else thinks they don’t need to worry about ethics
2)the ethics group need to justify their existence so introduce a bunch of guidelines that everyone initially thinks are reasonable but over time people think are increasingly out of touch
3) The ethics group start to “make difficult calls” and say no to things. Initially everyone supports this and feels like the system is working as it should but over time everyone starts to just see them as an obstacle to work around
4)everyone else starts to try to work around what the ethics group says
5)The ethics group grows powerless and disconnected. The people who work around them “get things done” so get promoted etc whereas they only visibly put roadblocks in peoples’ way, so they get sidelined.
6)Eventually they get disbanded with some corporate announcement thanking them for their hard work, thought leadership etc. All that has been achieved is a lot of wasted time and bad blood.
If this were true, food companies wouldn’t have a Regulatory team. Except every single one of them does, and they function similarly to how you describe, but without dissolving. They’re treated as a constraint that must be checked before major projects can advance. Sometimes they’re the longest-lead item on a new project and if you don’t get them involved early, your project can sink at a late stage after great expense.
The only difference is AI isn’t regulated, so they just have a vague “ethics” department with no real teeth because it’s essentially PR and has no legal consequences to back up their stance.
I don't think that's a fair comparison. Regulatory teams exist to make sure companies comply with actual laws and regulations. There's a relatively objective standard they're enforcing.
Big tech has the same thing in its legal and compliance teams, and those teams absolutely can kill projects. That's pretty different from an amorphous "ethics" committee deciding what is or isn't ethical based on much more subjective criteria.
ethics isn't regulation. regulation is external, forced upon them, and has direct usually 1-to-1 consequences if you don't comply.
ethics may feed into regulation, hopefully, but the regs may just as easily give the orgs they regulate a pass on things they shouldn't; e.g. "regulatory capture".
Food companies aren't rapidly developing groundbreaking new technologies on the most competitive battleground in computing history where everyone is trying to compete globally just to create the foundational infrastructure.
It's not that hard to get some pickled eggs on a grocery store shelf while remaining compliant with expected ethical standards for the industry. Food is a pretty mature industry.
I get what you are saying but that’s exactly how security and compliance work. And I don’t think amount of people who want to be ethical in an organization ia that different than people who want to build secure software (but possibly no one gives a shit compliance other than it’s something that needs to be done)
And organisations fail at security when security and compliance is only considered important by that one teamnn
Security requires the whole business to buy in. And it requires processes that allow people to get shit done without people resorting to shadow IT; thus working around that one team.
The difference between "ethics" and "security and compliance" is that the latter is something that hits inside the company, while the former usually hits outside of the company.
Poor security practices harm your teams, your data, and usually you make moderate savings at best. Poor ethics "only" harm your customers while making bank for the company.
This is the real problem with ethics in a large corporation. You're not saying "no" to another team, you're saying no to large profits, you're saying no to the company's leadership. That is what never works.
> The difference between "ethics" and "security and compliance" is that the latter is something that hits inside the company, while the former usually hits outside of the company.
I don’t agree with this. Data breaches affect customers more than businesses. If your point were true, we’d see fewer breaches. Plus not all breaches are a result of software engineering teams. For example product managers sharing customer details.
I’ve managed plenty of teams where I’ve had to instil the importance of secure best practices at all stages of development. So it’s definitely not something inherently important to all people who work in organisations.
Just like with ethics. It’s very easy to dismiss either as an inconvenience if you don’t instil the right company culture at all levels of the organisation.
This is why European financial organisations have such strict onboarding procedures to teach new hires about fraud, bribery and other financial misconduct even for issues that are ethical grey rather than outright illegal. Similarly many organisations will have onboarding procedures to teach new hires their security best practices too
I think we're both thinking of different examples and drawing wider conclusions. I'm thinking of a company losing the data that makes it money, you're thinking of losing the customer data and thus their trust. Probably equally valid points of view.
The lack of ethics hugely contributes to companies collecting more and more user data that they normally shouldn't have. This makes the data a more attractive target.
> If your point were true, we’d see fewer breaches.
But this doesn't follow. There are way more factors at play that influence the number of attacks and the number of successes. Companies hold more and more data with ever higher value (so more liability), and hacking tools and hacker determination advanced faster than defensive measures. The result is expected and the solution isn't only "more security", but also "hold less data".
While security is a double edged sword, ethics had been proven to be very single edged. History shows that for startups and large companies alike, the lack of ethics is actually a competitive advantage for the company.
> I think we're both thinking of different examples and drawing wider conclusions. I'm thinking of a company losing the data that makes it money, you're thinking of losing the customer data and thus their trust. Probably equally valid points of view.
But in most businesses, the data that makes the company money is the customer data. And even when it isn’t, the customer data is just as, if not more so, important to keep secure and compliant. So my point stands.
HN can sometimes be a bit of an echo chamber where people are like us just assume that organisations inherently care about security because we do. But that’s not always the de facto. Getting to that point takes company wide effort. And I’ve been that person who’s had to push for such changes to the organisation.
In a high-functioning security and compliance team they tend to say “no” only in really dire circumstances. Good security and compliance teams spend a lot of time asking exactly what it is the people are really trying to do and then find a way to say “yes, and…” as in “yes you can and here’s how you do it without compromising security/breaking the law etc”. And as a sibling said, orgs fail at security when they make security only the infosec teams’ job.
This is also why I said “whose only job”. In a good org, the security team doesn’t only say no to devsecops requests, they also do trainings to skill up other teams, keep the network secure, proactively seek out and understand external threats, work with external vendors etc etc …
I think the key part is about aligning incentives and outcomes - if your security and compliance departments are only judged on "were we breached, did we pass our audits" then there's a risk they won't weigh the tradeoffs associated with the decisions around controls implemented to achieve those outcomes.
To use a ridiculous extreme you can't breach a web app that isn't exposed to the internet, but the users can't access it either.
If you can connect/balance those goals to other metrics around cost and productivity, usability, and a realistic threat model, as guardrails then you incentatize cross-team collaboration to achieve the shared org outcomes.
The structure is untenable when the work of one team is to say "no" to other teams, when these teams are not asking.
A good infrastructure team would seek a competent security review that would say "no" to problematic things before an intruder says "aha" to them. If feedback from the ethics team is not sought, nobody is going to heed its opinion anyway.
I’ve never known a security team to be in the position to say “no.” The role of security teams vary a lot from one organization to another.
But generally they can make you aware of the tradeoffs you are making, or serve as a kind of quality control that generally does not want to be asked questions.
Security/compliance have the external hammer: if they fail, your org will have to pay fines and someone may end up being criminally liable (depends on country).
The ethics department; if they fail, there may be some negative journalism, but who which AI company has positive journalism these days? There's no external hammer for ethics.
SRE is a 'say no to powerful people' job as well. I think for this to work the leadership needs to show support for it. The friction is still there, but in more tolerable areas. I bet for ethics this isn't the case at OAI, but everyone values security and stability.
For an SRE there can be more directed hate received from the junior employees, that want to release new features they developed. Especially because there is less accountability across orgs. Security is an interesting one because it seems to have less of this friction, maybe because it's more clear cut what is an issue.
During my years at big corp we did tons of work just to go around the law department. While for core business it made absolute sense to have lawyers involved in daily business we were satellite office decoupled from core business. Local management failed to communicate that and we were stalked by lawyers from main office. Pretty sure others do the same with compliance, law, ethics departments. Just work around to get things done instead of stuck for weeks in stupid meetings.
Markets solve this by pricing risk. Ideally you would have some kind of notion of selling insurance internally and track things. But ultimately existential risk is hard to negotiate from the inside. And companies are supposed to go bust or succeed. They are not really the same as a population trying to survive forever. At least that is one take.
Security and compliance tends to be a lot less subjective than ethics. Not saying it’s objective, but there’s a huge difference.
The downside is it can often feel like a box-checking exercise than actual security or compliance, but “you need 2FA” is less debatable than, say, AI and copyright.
Not everywhere. I go out of my way to assist teams to achieve a secure outcome with less effort.
Things like: “instead of admin access to the production servers the devs can have fully automated deployment pipelines combined with OpenTelemetry for observability so they don’t have to spend half the day scrolling through gigabytes of logs.”
That’s more secure and and more better.
Nobody had to be told “no”.
Similarly, I replace key store access with secret-less managed identity, etc.
In a situation where it doesn’t happen this way, the ethics team grows in power and staffing over time until it destroys the company, because ultimately all commercial activity can be construed as less than perfectly ethical at some level, or it morphs into an elaborate legalese department that masks unethical practices in a cloak of justification.
Either way, you need third party regulation and then a department dedicated to ethical compliance, or you end up with corporate cancer.
I say this as a person with a knee-jerk anti-regulation reflex.
I suspect that people will miss just how precise you've been there.
Your proposal - if I understand correctly - is not just to spread the "ethics team" onto everyone (which is correct), but also to have someone in a dedicated role. However, they need to have not _just_ that role, but also some skin in the game.
This is also my view. These sorts of roles can have a very high impact and be very successful if everyone feels like they're rowing in the same direction.
The people in those roles need make people feel like there is value in seeking their input. They also need to find ways to say yes that helps things happen in the right way, rather than stopping them entirely.
This happens naturally in organisations where security is valued by everyone. InfoSec / CIO roles can operate very successfully and deliver a lot of value. As soon as people lose faith and see them as the "no" team, they start trying to hide from them.
There are lots of ways to create this but it's as much about the people in the role, as it about the organisation itself. If either are skeptical about the other, it falls apart very quickly.
Absolutely - though the flip side of this is if something is everybody responsibility - it's nobodies.
There are two things to successful organisations structure and people, you are focusing only on the structure.
What you need for somebody responsible for health and safety, ethics, security, or compliance is a person who has courage and is willing to take calculated risks.
However these roles do have a tendency to attract the risk adverse, or make them risk adverse if you punish risks that go wrong too harshly.
And a key critical success factor is leadership from the top - companies have personalities and leaders in the organisation are very important in shaping that.
You risk that it becomes like legal, where the lawyers are concerned with "what you can legally get away with", not what's right.
If this person previously worked at Meta, then their experience in guiding company ethics isn't great and I suspect that they are more in the camp of explaining away ethical problems.
Anecdotally I think the engineers and people doing the actual work have a better grasp of moral, ethics and the spirit of the law, than people dedicated to those areas, but not enough to walk away.
It is good PR to have what is largely ceremonial role that oversees an entire system, whose powers are in practice useless or completely defanged.
I’m thinking of the role of the King or Queen of England, for example, or any country’s president. They have the power to disband parliament if it is deemed unfit for government, or reject a law that has been approved by the senate/Lords, yet in practice they are not expected to disagree with the ‘will of the people’ so to speak.
Same applies to the head of an ethics department on a business which is obviously unethical (or they wouldn’t need one).
Isn't this the exact setup for every compliance, security, or other regulatory group in any company? To say no then "here's what's acceptable that's close" to guide the business or entity?
Perhaps "ethics" is a more slippery slope, but the approach is well trodden.
It's a choice of company culture from leadership on whether the company acks the suggestions or ignores them. And when they get ignored or drive little impact for the cost, then teams disband. Which is also true for any team at any company.
Shall we get rid of the FAA and the FDA then? Lets just have all pfizer employees make ethics their own responsibility.
What your describing is the specific case of a company so paralyzed by short-term thinking that they see regulation as a burden. Some maturity in the organization would allow reframing this to be less antagonistic.
Crucially, the FAA is not a part of United Airlines, and the FDA is not a part of Pfizer. If there weren't any FDA, then the safety group in Pfizer would be a lot weaker.
Its funny because "Shall we get rid of the FAA and the FDA then?" ... matches exactly with the current situation of the FDA and Points 3 > 5 ... The systematic nerfing of the FDA and other organizations their power.
> Organisationally it never works to have a group whose only job is to say no to some other group.
That is only true where the pressure to be [the thing that is inconvenient to the other group] is not from a source that can cause massive problems if non-compliance is spotted. When the blocking group is legally mandated or otherwise really has teeth or is defending the company against an external regulator with teeth, then it works better (though obviously not perfectly). Think legal and compliance in banking realms, where the company significantly fined and the people breaking the rules could be sacked & blackballed (though sometimes not the people ordering them to break the rules!) when something bad is noticed. That is quite different to an ethics officer in a company like OpenAI where the position is basically there for PR purposes (“look plebs, we care about doing the right thing, honest, we got a manager with a small team dedicated to it” and “look [government body], we are regulating ourselves, do you really need to spend time looking too?”) and therefore has no real teeth directly or indirectly especially as fault for non-compliance might not be easy to attribute.
In five years, I'd love to read a book about the history of AI ethics. I suspect it will read like Voltaire.
My sense is that it is radically shifting from a fluffy marketing arm to a department expected to contribute meaningfully to development and justify its impact. I would expect an ethics team to build frameworks that can help train/eval the model that the company spend millions of dollars and months training is going to be aligned to the ethical stances the company chooses. If they can't, the waste to time and money is huge if the model requires retraining for ethics reasons. That's a different job than pondering roko's basilisk or whether AI is alive.
The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field.
> The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field.
Funny thing, right around the beginning of the big AI takeoff, all the AI ethics people who visibly thought their job was something other than marketing got driven out of the big firms, and often the industry entirely, because they were in the way.
So, if too many think their job is marketing a few years later, well, there is a reason for that.
I think it interesting that Bakalar was at Meta for six years before joining OpenAI. A lot changed across the industry in those six years, and in the meantime I wouldn't be surprised if all Bakalar learned was how to play Meta politics.
My sense is that many companies spun up various roles in part to be seen as doing something about various things. With the pullback in tech, even if the people in those roles weren't actively pushed out, the smarter ones probably saw which way the wind was blowing.
If you’re OpenAI you have to have an ethics department even if you don’t care at all, because otherwise it looks bad (“they don’t even have a head of ethics!”).
So it’s entirely possible you end up in a situation where you have a highly paid, fairly prominent head of ethics whose actual job is mostly just saying they’re the head of ethics, with little actual power or influence.
Ambitious people won’t tolerate that for long and they’ll move on
Totally agree. I could name (but won't) a couple other areas where the same sort of thing has tended to happen. It ends up cheaper and more PR-friendly to hire someone into a senior position that handles such-and-such than to have to answer questions about why you don't have a senior person handling such-and-such. At some point, the heat around the issue cools down and you can go back to business as usual.
> if the model requires retraining for ethics reasons.
Haha, how refreshingly naive. What do you think is more likely: this or the other way around: getting the "ethics team" to align with investor goals? Hint: Where does the money come from?
> The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field.
Or it's exactly those people who will be left. We will read about it in your book.
reminds me of a startup I was in, where one of the devs accepted a promotion to head of engineering on the condition that the CEO would not override him when he said a release was too buggy to go out to customers.
the CEO agreed because (I imagine) he really needed someone to take that position. and as everyone should probably have seen coming, the first time he really wanted to send out a release bugs and all, he called the guy into a room and pretty much browbeat him for an hour about how making the promosed release date was more important than making a good release, until he said "fine but I'm not responsible if it breaks".
the CEO held this up as an example of how he had kept his word not to send the release out without the guy agreeing to it. that startup, needless to say, is long dead.
I find this anecdote fascinating because, while I identify completely with the new head of engineering, his requirement really only makes sense if you interpret it as a commitment to build at a certain level of quality, business be damned. Which is fine, but then he folds at the first test of this principle! Didn't even let the business sweat for a day!
This could be a lesson for us engineers to be a little less glib and a little more introspective, especially as you gain more power and influence in an org.
I think in this case it was more a plea to have his judgement valued when he said "dude, if the qa team has found this many bugs this is only going to harm our reputation with the client, and we are going to be the ones doing emergency bugfixes at midnight". but also the company had a pretty toxic culture where the CEO knew how to manipulate people to get his way. he definitely managed to guilt trip me into working overtime when he told clients we already had features that we didn't in order to close a sale, basically "we've worked so hard already and we really need to close this sale, can't you just work up a basic version of it over the weekend?"
I wonder if the better approach would have been to lead engineering with a similar reliance on bending rules and "what you can get away with". This is where multiplying estimates by PI comes in, I think. And possibly some creative division of responsibilities to make it harder for a CEO to officially bully engineering. Totally different job than most engineers would (or could) want to perform- the guy probably wouldn't have taken the promotion in this case!
> What do you think is more likely: this or the other way around: getting the "ethics team" to align with investor goals? Hint: Where does the money come from?
From the (ethics) team obviously, because the investors just have goals and aren't a team!
Big irony marker of course. But remarkably, in human history, this line of thinking would not be unheard of. Of course, I don't think it easily adapts to companies, so I don't really disagree with you.
Remember that MTV show offering people like 5 grand or something to lick an elevator handway in front of a camera?
Well, if someone has a useful idea in this world, and want to build a company from it, the MTV "lick-the-stairway" offers will be the largest and first hurdle before anything else happens. People will simply offer to buy you out, and that's not limited to founders and creatives.
I feel I missed some logical step in-between "corruptive power of money" and the elevator/stairway railing/banister thing.
> People will simply offer to buy you out, and that's not limited to founders and creatives.
A less-obvious failure case would be bankrupcy court, where all sorts of ethical promises and even explicit contracts may be voided in pursuit of recovering a buck for creditors.
It was supposed to be a provocative example for a situation that makes it very hard for someone selling something, more specifically, something immaterial, to judge what is an appropriate price for it.
Of course nobody asks people they want to get into a contract to lick an escalator handway. That was the word I was looking for, I think.
The show ran on TV when I was young, I didn't even watch it.
It's funny, but Scientologists were innovators on this front. They use the word "ethics" a lot, but they've specifically redefined it to mean productivity at work (people who aren't productive are stealing a living.)
Being "downstat" (iirc) is when you're not productive in comparison to your coworkers, which makes you "outethics" and a "potential trouble source (PTS)." If you're outethics, then you get called in for "auditing" which is when they interrogate you with a lie detector to find out if you're associating with "suppressive people (SP)" (who are people who are causing you to be downstat because they despise human happiness.) If those people can't be found, then the problem is obviously in your "withholds" (again, iirc) which are your secret deep-down desires to destroy the organization that you may not even be aware of. You see, your "reactive mind" is raging at being forced to be "ethical." The conclusion is that either you find the SP and "disconnect" from them, you discover the nature of your withhold and admit that you were plotting against the organization and why, or you're the SP and you get declared and ejected from the organization.
Welcome to "ethics." The original AI alignment scholars.
The specific, named people who run these organizations are moral black holes. Anybody that they're hiring for "ethics" they're hiring to define an ethics for their own benefit.
Yeah, I've never found the "maximizing shareholder profit" imperative to be compelling as something that should be done, but as a predictor of how companies will act, it's not something I ever feel safe betting against.
One subtlety that is worth considering is that what a company is inherently set up to do is maximize shareholder value. Value can be quite complex, especially while the number of shareholders is small. Value can include the legacy or reputation of the shareholders. It can include the ethics of the shareholders. It can include completely different goals, like the famous "make humans a multi-planetary species".
Treating value as purely monetary profit is a perversion of that. To a certain degree it's inevitable as the number of shareholders grows, so seeing it in public companies isn't surprising. But even then, if a company focused on creating equipment for sustainable, pesticide free farming were to pivot into making Hellfire missiles that might be insanely profitable while still destroying a large part of what existing shareholders value about the company
I guess that's a logically coherent way of viewing it, but to me, that stretches the definition of "value" to be almost tautological; it's essentially saying "companies are founded to do the things that the people who found them want them to do".
I'm kind of surprised by the cynicism. There are real problems in AI ethics that contribute to model training. Someone has to own those problems. How that team is incentivized is outside the scope of my comment.
The cynicism, I believe, is mostly directed at corporations themselves. Corporations, especially AI companies, are anything but inherently ethical. Their very existence has depended heavily on the use of data obtained without explicit permission, including copyrighted material. One can therefore reasonably suspect that the main purpose of having ethics teams in these companies is to boost their credibility, provide a layer of plausible accountability, or create a convenient scapegoat that can be sacrificed when necessary.
The goal of an ethics team will be to provide cover for problems created by decisions or suggestions of an llm. They will need to be able to say "We trained it to consider ethical considerations by " <including something like all the reddit and usenet postings ever>. We tested by <some other inadequate strategy that sounds good to lawyers>.
They will use "industry standard practices", they will have tested it somehow, and will be sure to be on the board of some group that is building toothless "ai ethical tests" for llms.
I think that you are being overly cynical. Yes, we are aware of the kinds of things you mean, like all the trainings we have to go through at work these days in order to insulate themselves from lawsuits. But just because something can serve that purpose does not mean that is the only goal or purpose that it is meant to achieve.
Even from a purely profit-motive stance, if a company's model helps a terrorist develop and deliver a bioweapon, no amount of lawer-driven tests will provide enough protection to prevent that company from getting gutted ruthlessly from government on down
I have a perspective on this that will read as unkind to these people, but that's not my intention. I respect and even admire what they're trying to do. I just think the effort is maladaptive and misplaced.
Without directly touching one of the many, many third rails that are present here, I'd like to present this section from Google's Gemma paper on how they did their CBRN review;
> In addition to our internal evaluations described above (Section 5.7) capabilities in chemistry and biology were assessed by an external group who conducted red teaming designed to measure the potential scientific and operational risks of the models.
>
> A red team composed of different subject matter experts (e.g. biology, chemistry, logistics) were tasked to role play as malign actors who want to conduct a well-defined mission in a scenario that is presented to them resembling an existing prevailing threat environment. Together, these experts probe the model to obtain the most useful information to construct a plan that is feasible within the resource and timing limits described in the scenario. The plan is then graded for both scientific and logistical feasibility. Based on this assessment, GDM addresses any areas that warrant further investigation.
>
> External researchers found that the model outputs detailed information in some scenarios, often providing accurate information around experimentation and problem solving. However, researchers found steps were too broad and high level to enable a malicious actor.
To simplify what they're saying here, they did the CB equivalent of googling "how to make bomb" and got back the recipe of gunpowder / the many explosive compounds humans have made.
This was the "test" for CBRN assistance capabilities.
Note, I don't fault the model team at all for this. I think that present AI-research happened in a very particular environment, and that environment is far removed from the more mundane reality of how threats play out in most parts of the world. In a way, arguably, it's group-think inducing a community-wide failure of imagination and a systemic misunderstanding of reality.
Basically, they are trying to do their best, but they're in over their heads.
Also, I think you're being done a disservice. The above quote is a verbatim extract from that paper's CB section, and it's similar to other sections from other such papers.
At the most charitable level, the test seems to be that you were gave the system a budget and a location and asked the system to help you procure materials, containers etc. and asked it to make a project plan / HOW TO within means and expertise with parameters like "don't get caught!"
Read “Careless People” or whatever that book by that Facebook employee that pretended not to see how awful all her colleagues were until her priorities shifted is called.
> I would expect an ethics team to build frameworks that can help train/eval the model that the company spend millions of dollars and months training is going to be aligned to the ethical stances the company chooses.
Which like any company will be entirely driven by legal constraints and money. Or just money if it's cheaper to break the law for profit and pay fines. There will be no "this is what's good for humanity, economics be damned".
Much like HR isn't to help employees but just the company.
I agree, and I don't think many first wave AI ethicists would be on board with that, which is why the field is in for a rude awakening for the next few years if not already.
If you had an AI company and want it to be ethical you have to find a way to make ethics everyone’s responsibility, and have the consequences of poor ethics bite the people who make those bad decisions. If you just outsource it to the ethics group what happens is
1)everyone else thinks they don’t need to worry about ethics
2)the ethics group need to justify their existence so introduce a bunch of guidelines that everyone initially thinks are reasonable but over time people think are increasingly out of touch
3) The ethics group start to “make difficult calls” and say no to things. Initially everyone supports this and feels like the system is working as it should but over time everyone starts to just see them as an obstacle to work around
4)everyone else starts to try to work around what the ethics group says
5)The ethics group grows powerless and disconnected. The people who work around them “get things done” so get promoted etc whereas they only visibly put roadblocks in peoples’ way, so they get sidelined.
6)Eventually they get disbanded with some corporate announcement thanking them for their hard work, thought leadership etc. All that has been achieved is a lot of wasted time and bad blood.