Sunday, 27 March 2016

The dark side of digital finance: On financial machines, financial robots & financial AI

Note: I published a shorter version of this in Nesta's magazine The Long+Short as You Are the Robots. This is a modified and extended version, published under Creative Commons

A banker in 1716 had two main tools: a ledger book and a quill pen. A customer – perhaps a prominent carpenter – would enter a branch, request a withdrawal or make a deposit, and the banker would make a careful note of it within the ledger, editing the customer's previous entry to keep authoritative score of exactly what the bank promised to them.


Fast-forward to 2016 and we’ve entered into a world no longer dominated by tools, but by machines. The crucial difference between a tool and a machine is that the former relies on human energy, while and the latter relies on non-human energy channelled via a system that replicates - and accentuates - the action of a human using a tool. The carpenter is now a furniture corporation using computer-programmed CNC cutters. Likewise, the bank that keeps score of that company’s money runs humming datacentres with vast account databases. These are digital equivalents of the old ledger books, drawing upon fossil-fuel generated electricity to write and hold information as magnetised atoms on hard-drives.


We call the process of moving from manual tools to machines automation, and it appears in various forms within everyday financial life. The ATM, for instance, is an automated version of the bank teller of old who would have to exert energy to check your account, hand you cash, and alter your accounts. I use an interface to interact with this ATM, which gives me some form of control, but only within the inflexible rules of whatever it will allow me to do. This actually requires energy on my part, so while the machine seems to ‘do things for me’, the process also seems to be ‘self-service’.

Automation is creeping into more and more of personal finance. The glossy adverts of the financial marketing industry put an appealing spin on the future world of contactless payment, branchless banking and cashless society. They focus the mind on problems that are apparently being solved through new technology, but they simultaneously divert attention from the dark side of the automated financial regimes that are emerging around us. To get to grips with these processes of automation - and the sub-field of 'digitisation' - we first need to establish some definitions of machines, robots, and algorithms.

Financial machines vs. financial robots

Machines tend to require us to manually activate them towards a singular repeated action that they do no matter what, like the way a kettle always boils water if I manually push the ‘on’ button. The ATM is a multi-function machine that can do different things if I push different buttons on the interface, like ‘give me £30’ or ‘show me my balance’. It doesn’t, however, seem to ‘make decisions’ or have any ability to autonomously react. To make it feel like a robot, it must show some nominal agency to make decisions based on external information.

To understand what a financial robot looks like, we need to sketch some general characteristics of robots more generally. We might think of a traditional robot as a system comprised of four parts:
  1. Body: An assemblage of mechanical parts
  2. Mind: An algorithmic ‘mind’ that can compute or analyse information
  3. Senses: Sensors that can detect external data
  4. Energy source: For example, electricity
The traditional robot might take in data from sensors and compute it through an algorithmic mind that can activate the mechanical body, provided there is electricity. For example, a robot could be a vacuum cleaner (mechanical body) that receives data from photocell sensors (senses) to be processed through an algorithm (mind) to calculate its position, which in turn sends orders for the body to move around the room, thereby 'autonomously' vacuuming your lounge by 'making decisions'.  

Importantly, though, it may not be necessary to include the mechanical ‘body’ part at all. A robot might simply be a software-based algorithmic ‘mind’, taking in data and sending orders to other entities to act out its ‘will’. We might call this an algo-robot.

Let’s consider an Excel spreadsheet model that is used to estimate the fair price of a financial instrument like a share. A person armed with a pen and pad might take hours or even days to go through the relevant data and do the calculation manually. The spreadsheet model on the other hand, directs the electricity coursing through the hardware of a computer to do the same calculation in a fraction of the time. This is a financial machine, automating manual human calculation processes.

To make this into a robotic system, though, we must allow it to receive perceivable external data – such as a price feed from the London Stock Exchange – and allow it to process the data through its ‘mind’ of algorithmic formulas, and then give it the ability to make executive decisions based on its calculations (like the ability to send buy or sell orders back to the stock-market). And, voila, this is precisely what algorithmic automated trading is. The spreadsheet model has turned into a trader algo-robot. From this point the algorithmic coding can be developed into more ‘human’ forms, for example by equipping the robot with machine learning capacities and ‘evolutionary algorithms’ that can adapt to changing circumstances.

The algo-robotic managers of digital finance

‘Algo-robotic’ systems are particularly adept at accumulating power. Unlike the simple machine that offers static options via an interface, an algo-robot - or a series of linked algo-robots - have a greater ability to react in multiple ways in response to multiple data streams, and therefore to organise and co-ordinate. This trait makes senior corporate management warm to them, because, after all, reacting and co-ordinating are core elements of what a manager does.

The old hierarchy within a corporation was one where owners used managers to co-ordinate workers and machines. This gave rise to the traditional battles between owners and managers, managers and workers, and workers and machines. The emergent hierarchy is subtly different. The owners – often a disparate collection of distant shareholders – grant power to high-level management, who increasingly use algorithmic systems as ‘middle management’ to organise their workers and more basic machines.

And this is where we see the changing conception of the robotic system’s ‘body’. Rather than being a mechanical assemblage with an algorithmic ‘mind’, the robot could be an algorithmic mind co-ordinating a ‘body’ constituted out of ordinary employees, who increasingly act like machine parts. Think about the Amazon deliveryman driving the van to act out an order sent to him by an algorithm. This ‘body’ doesn’t even have to be constituted by the company’s own employees, as in the case of self-employed Uber drivers co-ordinated by the Uber algorithms.

These arrangements are often difficult to perceive, but algo-robotic systems have been embedding themselves into everyday forms of finance for decades, not necessarily 'taking over control' but often creating a hybrid structure in which manual human actions interact with automated machine-robot actions. For example, the investment bank trader might negotiate a derivatives deal over the phone and then book it into a partly automated back-office system.

The quintessential example, however, is the retail bank branch. You can talk with employees behind the Barclays counters, but often they are just there to enter data into a centralised system that tells them how to deal with you. To some degree these employees have agency – the ability to make quasi-autonomous decisions – but the dominant trend is for them to become subservient to the machinic system they work with, unable to operate outside the bounds set by their computer. Indeed, many bank employees cannot explain why the computers have made the decisions they have, and thus they appear as the human face put there to break the news of whatever the algorithm has decided. We might even say they are a human interface to an otherwise algo-robotic system that is accountable only to the senior corporate management, who you will never deal with.

NOTE: If you have enjoyed this so far, you might like my book

From hybrid systems to self-service digital purity

But, 'human interfaces' like that are actually quite costly to maintain. People are alive, and thus need food, sick leave, maternity leave and education. They also have a troublesome awareness of exploitation and an unpredictable ability to disobey, defraud, make mistakes or go rogue. Thus, over the years corporate managers have tried to push the power balance in this hybrid model towards the machine side. In their ideal world, bank executives would get rid of as many manual human elements as possible and replace them with software systems moving binary code around on hard drives, a process they refer to as 'digitisation'. Corporate management is fond of digitisation – and other forms of automation – because it is a force for scale, standardisation and efficiency – and in turn lowers costs, leading to enhanced profits. 

The process is perhaps most advanced in the realm of electronic payments, where money is shifted with very little human action at all. Despite recent talk of the rise of digital currencies, most money in advanced economies is digital already, and tapping your contactless payment card sets in motion an elaborate automated system of hard-drive editing that 'moves' your money from one bank data-centre to another. This technology underpins talk of a future 'cashless society'. Bouncy startups like Venmo and iZettle have got into the payments game, adding friendly new layers to an underlying digital payments infrastructure that is nonetheless still dominated by the banking industry and credit card networks.

In the case of retail banking, an ideal situation for banks might be to get rid of the branches altogether, and to push for a world of ‘branchless digital banking’. This generally means slowly dismantling, delegitimising and denaturalising branches in the public imagination, while simultaneously getting people accustomed to 'self-service'. Indeed, many banks are cutting branches, and many new forms of financial services are found only online, like digital banks Fidor and Atom. Digital banking startup Kreditech claims that bank branches won't exist 10 years hence, "and neither will cost-intensive, manual banking processes". "We believe algorithms and automated processes are the way to customer-friendly banking," the startup declares confidently. 

Such digital banking is but one strand in the digital trajectory. Digitisation is starting to be applied to more specialist areas of finance, too, such as wealth management. Wealthfront, for example, now offers automated investment advice for wealthy individuals. In their investment white paper they state that sophisticated algorithms can "do a better job of evaluating risk than the average traditional advisor".

Digital systems like Wealthfront are often promoted as cutting out the middleman – assumed to be human, slow, incompetent and corrupt – and therefore as cutting costs in both money and time. Some startups use this to build a narrative of the 'democratisation of finance'. Quantopian, a system for building your own trading algorithms, comes with the tagline: "Levelling Wall Street's playing field". Robinhood draws on the name of the folk hero to pitch their low-fee mobile stock-trading system. 

It seems uncontroversial that these systems may individually lower costs to users in a short-term sense. Nevertheless, while startup culture is fixated upon using digital technology to narrowly improve short-term efficiency in many different business settings, it is woefully inept at analysing what problems this process may accumulate in the long term. Payments startups, for example, see themselves as incrementally working towards a 'cashless society', a futurist buzzword laden with positive connotations of hypermodern efficiency. It describes the downfall of something 'old' and archaic – cash – but doesn't actually describe what rises up in its place. If you like, 'cashless society' could be reframed as 'a society in which every transaction you make will have to be approved by a private intermediary who can watch your actions and exclude you.' It doesn’t roll off the tongue very well, and alarms the critical impulses, but nevertheless, that’s what cashless society would bring.

Forcing the 'inevitable progress' of digital finance

Part of the reason for the pervasive acceptance of these developments is the deeper ideological narrative underpinning them, one which is found within the tech industry more generally. It is the idea, firstly, that the automation of everything is inevitable; and that, secondly, this is 'progress': a step up from the inefficient, dirty services we have now. In this context, questioning the broader problems that might emerge from narrowly useful automation processes is ridiculed as Luddite, anti-progress or futile.

Of course, ‘progress’ is a contested term. If you’re cynical, you may see it as shorthand for ‘the situation an organised set of commercial interests view as desirable in the short-term’. It doesn’t necessarily mean ‘the thing that would be good for the broader public in the long term’.

Indeed, it is apparent that many people don't respond to 'progress' in the way they're supposed to. We still find people insisting on queueing to use the human cashiers at big supermarkets like Tesco, rather than diligently queueing up for the automated checkout. Likewise, we still find people stubbornly visiting the bank branches, making manual payment requests; even sending cheques.

Perhaps this is because there is something deeply deadening about interacting as a warm-blooded individual with a soulless automaton trying to sound like a human. The hollow fakeness of the cold clinical checkout voice makes you feel more alone than anything else, patronised by a machine clearly put there to cut costs as part of a faceless corporate revenue circuit.

The ongoing challenge for corporate management, therefore, is how to push automation while keeping it palatable. One key technique is to try to build more 'human-like' interfaces, and thus in London we find a hotbed of user-experience (UX) design firms. They are natural partners to the digitisation process, combining everything from ethnographic research to behavioural psychology to try to create banking interfaces that seem warm and inviting.


Another key technique is marketing, because people often have to be 'taught' that they want something. In the case of contactless payment on the London Underground, the Mayor of London, Barclaycard, Visa and the Evening Standard have formed an unholy alliance to promote Penny for London, a thinly veiled front-group to encourage people to use the Barclaycard-run contactless payments system rather than those ancient Oyster cards. Sports stars like Jessica Ennis-Hill and Dan Carter have been co-opted into becoming the champions of automated finance. Signs have been popping up proclaiming 'contactless is here', as if it were something that people were supposed to be waiting for. These subtle hegemonic messages permeate every financial billboard in the city.

The dark side of digital finance

One key to developing a critical consciousness about technology is to realise that for each new innovation a new trade-off is simultaneously created. Think about the wonderful world of digital banking. A low-level bank branch manager might be subservient to the centralised system they work for, but can also deviate subtly from its rules; and can experience empathy that might override strict economic 'rationality'. Imagine you replace such an individual with an online query form. Its dropdown menu is the digital equivalent of George Orwell's Newspeak, forcing your nuanced, specific requests into blunt, standardised and limited options. If your problem is D, a system that only offers you solutions to A, B, or C is fundamentally callous. A carefully constructed user complaints system can build an illusion of accountability, while being coded firmly to bias the interests of the company, not the user.

Indeed, if you ever watch people around automated self-service systems, they often adopt a stance of submissive rule-abiding. The system might appear to be 'helpful', and yet it clearly only allows behaviour that agrees to its own terms. If you fail to interact exactly correctly, you will not make it through the digital gatekeeper, which – unlike the human gatekeeper – has no ability or desire to empathise or make a plan. It just says 'ERROR'.

This turns out to be the perfect accountability and cost cushion for senior corporate management. The responsibility and energy required for dealing with problems gets outsourced to the users themselves. And lost revenue from unhappy customers is more than compensated by cost savings from automation. This is the world of algorithmic regulation, the subtle unaccountable violence of systems that feel no solidarity with the people who have to use it, the foundation for the perfect scaled bureaucracy.

So, in some future world of purely digital banking we find the seeds of a worrying lack of accountability and an enormous amount of user alienation. The loan you applied for online gets rejected, but nobody is there to explain what hidden calculations were done to reach that decision. To the bank management, you are nothing more than an abstract entity represented by machine-readable binary code.

So where is the financial AI?


Of course, the banks don't want you to feel like that. In the absence of employees, they will have to use your data to create the illusion of some type of personally tailored service. Your historical interactions with the system will be sold back to you as a ghostly caricature of yourself, fed through the user-experience filters. And it is here that we find the emergence of new forms of financial artificial intelligence.

Let’s return to the earlier – somewhat blurry – distinction between machines and robots: robots are essentially machines that take in data from sensors and process it through an algorithmic 'mind' in order to react or 'make decisions'. Likewise, there is a blurry line between robots and artificial intelligence. At its most unambitious, AI is just a term for any form of calculation done by robots. It really comes into its own, however, when referring to robots that have adaptation and learning capabilities which allow them to show creativity and unexpected behaviour. Rather than merely responding to your actions or to external stimuli, the system begins to predict things, offer things, make suggestions, and do things without explicitly being asked to do them.

Imagine, for example, an ATM booth that uses facial recognition technology to identify you as you approach and make suggestions to you. Notice how the power dynamic changes? With a normal ATM I am still an active body, choosing to trigger the machine via the interface. In this new scenario, though, I’m a passive body who triggers the machine without any explicit conscious action on my part. It seems to 'take the initiative' and to direct me. It's only when we start to feel this as a power dynamic that we start to get closer to the feel of AI. The more you move towards AI, the more you feel increasingly passive relative to the robot (a passivity that is beautifully captured in this video).

Consider the customised ads Google feeds to us. We don't actively try to make them appear, yet it's still our actions that trigger the system to target us with specific information. That’s more like AI. There are many scenarios where this process could creep into finance, from machine-learning trading algorithms to creepy health insurance contracts that shift their prices according to your mobile payment data. "I see you paid for two chocolates today Brett. I will raise your premium."

But this can go beyond a single machine. Just like a robotic system may actually be constituted by an algorithmic 'mind' that coordinates a 'body' of people – like Uber drivers acting out the will of their invisible algo-boss – so the body of an AI may be fragmented, decentralised and hard to perceive. It could be a network of interacting algo-robotic systems that direct the actions of people who are unaware they are triggering the system. No individual node may be in control, but people may collectively become locked into reliance upon the system, pulled around by forces not immediately apparent to them, being manipulated by their own data. The AI could be a ghost in the collective machine, the manipulative 'invisible hand' in a technologically mediated market.

Don't panic, but don't not panic either

When thinking about the future of digital finance, the issue is not necessarily whether these services are narrowly useful to an individual. Sure, maybe the contactless card is cool if I'm in a hurry and maybe I can get a decent deal from the AI insurance contract. Rather, the issue is whether they collectively imprison people in digital infrastructures that increasingly undermine personal agency and replace it with coded, inflexible bureaucracy; or whether they truly offer forms of 'democratisation'.

It is easy to overhype these scenarios, though, because while it is true that payments, trading and retail banking are increasingly subject to automation, finance as a whole may not be especially amenable to it. Large loan financing decisions, complex multistage project-financing deals, exotic derivatives and other illiquid financial products cannot easily be standardised. They require teams of lawyers and dealmakers hashing out terms, conditions, and contingencies. Finance is an ancient politicised art of using contracts about the future to mobilise current action, and the dealmakers cannot easily be replaced with algos.

Furthermore, attempts to create more advanced and intuitive automated systems frequently fail. Semantic analysis algorithms – designed to read text – are terrible at understanding irony, sarcasm and contextual ambiguity within language. They may create feedbacks that thwart their own purposes, as in when people learn to game a credit-rating algorithm. High frequency trading falls apart under its own excesses and becomes less profitable. And there are customer backlashes: Metro Bank, the first new high-street bank in Britain for 150 years when it launched in 2010, has grown precisely because of its explicit focus on human-centred branch banking.

Nevertheless, it would be unwise to ignore the fact that the corporate trajectory is very much towards trying to automate as much as possible, and people need to come to terms with both the implications of this, and the vested interests behind it. It is not a neutral, 'inevitable' process. There are particular parties who seek it out. Take a moment to investigate who is on the board of Penny for London, that altruistic charity that insists contactless payment is a great way to help those in need. It includes hedge fund mogul Stanley Fink, and previously included the ex-CEO of Barclays, Bob Diamond.

So how should one respond? One approach is to ride with the technology, rather than to resist it. In intellectual leftwing circles the accelerationist sect advocates an embrace of automation, standing against sentimental calls for more human, local systems. It's an abstract position, founded on beliefs that automation will create conditions ideal for the downfall of capitalism. At some point it intersects with the cult of the Singularity, popular among evangelical tech entrepreneurs and transhumanists.


The ideological ambiguity is perhaps most acute in the emergent field of blockchain technology. Such systems potentially offer a way for strangers to freely interact with each other without central human intermediaries getting involved in the process. They may use blockchain systems to issue shares, enter into insurance contracts and form digital co-operatives, but the systems are underpinned by an extreme version of automation, one that is essentially autonomous. Indeed, the deep-level mission of projects such as Ethereum, a decentralised platform for 'trustless' transactions, is the replacement of human systems of institutional trust – like the legal and political systems that normally underpin all contracts and markets – with automated ones apparently detached from the human ambitions of those who historically have run such systems ('the politicians', 'the regulators', 'the bankers'). Libertarians long for an automated 'Techno-Leviathan' to replace the human sovereigns we have now, but it is a big question as to whether such automated systems truly provide a more 'democratic' infrastructure for interaction.

More down-to-earth are those who want to allow more creative interaction with the existing digital infrastructure. Take the Open Bank Project, for example, which wants to facilitate third-party customisation of digitised banking processes by opening up bank APIs, in the same way that independent developers might build third-party Twitter apps that draw data from Twitter's API.

And, finally, we have those who authentically seek to harness digital technology to bypass and challenge the standard economic rationality of large scale, short-term profit-seeking financial beasts, taking advantage of the lower startup costs of a digital setting to promote peer-to-peer finance, alternative currencies, crowdfunding platforms and non-monetary sharing platforms.

So, the scene is set. One thing is for sure: if the future of banking is going to be digital, we want it to be populated with those who value the deeper tenets of open source philosophy. Otherwise we could be left with increasingly alienating, exclusive and unaccountable financial surveillance states, presiding over increasingly passive and patronised users.

If you'd like to support my ongoing Creative Commons writing, please consider buying me a virtual beer


  1. Excellent overview of current tech issues in finance and the AI trend in general

  2. Great article... Just a few comments....

    ...Corporate management is fond of digitisation – and other forms of automation – because it is a force for scale, standardisation and efficiency – and in turn lowers costs, leading to enhanced profits. 

    If Barclays had a robot human like a cashier. More people than you may think would prefer that option. Increasing numbers of people actually prefer non human interfaces.

    Companies act to increase their survivability - profit is just one metric. If their competitor automates to provide a better service at lower cost, they may themselves go out of business altogether. Or upper management might lose their jobs. It's not always just cold hearted profit that drives automation.

    ....every transaction you make will have to be approved by a private intermediary who can watch your actions and exclude you.

    With private intermediaries and free market competition, you can just change providers. However, should a centralised public body ever excludes you, there is little hope left.

    ...progress..... ‘the situation an organised set of commercial interests view as desirable in the short-term’. It doesn’t necessarily mean ‘the thing that would be good for the broader public in the long term’.

    Each act of 'progress' must be incrementally tested in the market for people to choose. It almost always leads to lower costs, higher quality or freeing more time for customers. These are universal values people treasure. Any 'progress' in the opposite directions would quickly go out of business. It's not about profit, it's about survival. Becoming the best provider out there.

    1. Glad you like it!

      I think where we disagree is on three basic assumptions

      1) A banking-specific point: You seem to think an effective market is present in retail banking and yet to me, retail banking (at least in places like the UK) is one of the most non-competitive markets in the world. If you approach the majority of people and ask them to give you a rational account of why they have chosen to bank with a particular bank, most of them cannot do it, because often there is no meaningful distinction between the banks and furthermore, they've often just been given accounts as teenagers that they never close. There is a saying that says 'people are more likely to get divorced before changing a bank account'

      2) Another banking point: You seem to think that there is an power symmetry between banks and individuals, and yet to me banks have hegemonic cultural power that far exceeds that of individuals. Moreover, they often act as collectives. The big banks reach consensuses among themselves about what constitutes innovation and then often just 'tell' people that this is what will happen. Sure, to some extent they have to keep up with each other, but the innovation is not driven from 'consumer demand' - it is driven from a cost-cutting drive then they then turn into consumer demand through various mechanisms

      3) A more general point on markets: You seem to think that people's short-term interests correspond with their long term interests, and I do not. Assuming there are equal power dynamics (which there are not) markets are excellent at playing to people's short-term interests (short-term interests include things like convenience, cost-savings etc.), but they are completely incompetent at accommodating longer term interests (for example, as mentioned above, the desire to not be under surveillance).

      There are many instances in life that show us that short-term interests do not correspond with long-term ones. For example, a student might desire to smoke weed during the day instead of studying, but while they might gain pleasure from their attainment of this short-term utility they screw themselves over in the long term as they fail their exam. Markets thrive on this short-term prioritisation

      Nevertheless, market fetishists will tend to make two claims 1) that the short-term interests of small disparate buyers are aligned with the short-term interest of big powerful producers and 2) the short-term interests of buyers and aligned with their long-term interests. I think both of these are pretty dubious

  3. Yes, I do agree with your points. Especially about choosing banks. I'm guilty of that exact behaviour myself.

    And you bring up another very interesting point about short term vs long term interests. I need to think a lot more about, as id never considered this aspect before.

    My initial reaction it is see that it's easy to exploit people's economic time preferences which likely correlate with iq. So markets likely predate more upon weaker minds to aggregate economic gains. Freedom is not a system to protect weak minds from strong. It's in fact, the very opposite.

    But people have their own aesthetic preferences and prejudices. The only alternative to freedom are coercions of various kinds. Fine if the coercionists are both wise and benevolent. Maybe Ai can take on this role for us?

    But I am an optimist. I think technology will level the 'wealth gap' so long as wealth does not confer power, which is more a political than an economic issue.

    Most people don't fear losing their privacy until it is too late. Like believing in a fiat currency to store value, until it doesn't and then it's too late. Everything in life boils down to luck. Not just random chance. But genetics and wisdom too. No one is truly culpable.

    It's also fair to say I'm likely a free market fetishist. I tend to equate free markets with personal freedom and dignity. Although I am certainly in favour of basic incomes so everyone can benefit by being free to make these choices.

    Anyway, thanks for your considered reply.

  4. The main advantage of utilizing machines as opposed to manual human intervention is the amount of time and effort the process would likely to save. The term human error would also be eliminated but that does not guarantee an error-free process at every transaction. System bugs could still create a mess which would eventually require human intervention as well to rectify the technical glitch.

    1. I don't think anyone denies that machines can save time and effort. Indeed, that is precisely what they're designed to do. Merely pointing that out doesn't somehow prove that there are no dark sides or tradeoffs involved in this process. That's what the article was about