A decade ago, deep-learning models began outperforming humans on a variety of tasks, including outperforming world champions board game players and outperforming doctors at diagnosing breast cancer.
These potent deep-learning models are frequently built on the artificial neural networks framework, which was first put forth in the 1940s and has since gained popularity. Using layers of interconnected nodes, or neurons, that resemble the human brain, a computer learns to process input.
Artificial neural networks have expanded with machine learning as a field.
Deep-learning models are now frequently made up of millions or billions of interconnected nodes arranged in many layers and trained on enormous quantities of data to perform detection or classification tasks. But even the academics who create the models don’t fully grasp how they operate because the models are so very complex. This makes it challenging to determine whether they are operating properly.
For instance, a model created to aid doctors in patient diagnosis might have correctly predicted that a skin lesion was cancerous, but it may have done so by concentrating on an unrelated mark that frequently appears in photos of cancerous tissue rather than the cancerous tissue itself. This is an example of an erroneous correlation. The prediction is accurate by the model, but for the wrong reason. It might lead to missed diagnosis in a real-world clinical situation where the mark does not show up on photos showing cancer.
How can one figure out what is happening inside these so-called “black-box” models when there is so much uncertainty around them?
This conundrum has given rise to a fresh and quickly expanding field of research where scientists create and test explanation methods (also known as interpretability methods) that aim to explain how black-box machine-learning models make predictions.
EXPLANATION METHODS: WHAT ARE they? Methods of explanation are either global or local at their most fundamental level. While global explanations aim to describe an entire model’s overall behavior, local explanation methods concentrate on explaining how the model arrived at a single prediction. This is frequently accomplished by creating a different, simpler model that matches the larger, black-box model and is presumably understandable.
But because deep learning models function in fundamentally nonlinear and complex ways, it is particularly difficult to create a global explanation model that is effective. According to Yilun Zhou, a graduate student in the Interactive Robotics Group of the Computer Science and Artificial Intelligence Laboratory (CSAIL), who studies models, algorithms, and evaluations in interpretable machine learning, this has caused researchers to shift much of their recent focus instead toward local explanation methods.
There are three major groups into which the most common local explanation techniques can be divided.
The very earliest and most popular kind of explanation technique is called feature attribution. The features that the model prioritized when it made a certain decision are displayed through feature attribution methods.
A machine-learning model uses features as its input variables, which are then used to make predictions. Features are taken from a dataset’s columns when the data is tabular (they are transformed using a variety of techniques so the model can process the raw data). On the other hand, for jobs involving image processing, each pixel in an image is a feature. The feature attribution technique would draw attention to the pixels in that particular X-ray that were most crucial for the model’s prediction, say, that an X-ray image reveals malignancy.
In essence, feature attribution techniques reveal what the model concentrates on when making a prediction.
You can determine whether a spurious association is a cause for concern using this feature attribution explanation. For instance, it will display whether the pixels in a watermark or an actual tumor are highlighted, according to Zhou.
A counterfactual explanation is a second category of explanation technique. These techniques demonstrate how to alter an input such that it belongs to a different class given an input and the prediction made by the model. For instance, the counterfactual explanation reveals what has to happen for a borrower’s loan application to be approved if a machine-learning algorithm predicts that she will be denied credit. Perhaps she needs a higher credit score or income, two factors that the computer utilized to make its prediction, for her to be accepted.
The advantage of this explanation style is that it explains precisely how changing the input will cause the decision to change, which could be useful. This explanation would, he claims, explain to someone who applied for a mortgage but didn’t obtain it what they should do to get the result they want.
Sample importance explanations are the third group of explanation techniques. This approach needs access to the data that were used to train the model, unlike the others.
A sample importance explanation will highlight the training sample that a model most frequently used to make a certain prediction; ideally, this sample will be the one that is closest to the input data. This kind of justification is especially helpful when one notices an apparent irrationality in a prediction. A specific sample that was used to train the model may have been impacted by a data entry error. With this information, the sample could be fixed, and the model could be retrained to function more accurately.
WHAT METHODS OF EXPLANATION ARE USED? To do quality control and troubleshoot the model is one reason for creating these explanations. For example, with a better understanding of how characteristics affect a model’s decision, one may spot when a model is malfunctioning and take action to rectify it, throw the model out and start over, or both.
Exploring the use of machine-learning algorithms to find scientific patterns that humans haven’t yet found is another, more recent topic of inquiry. For instance, a cancer diagnosis model that performs better than clinicians may be flawed, or it may be detecting hidden patterns in an X-ray image that represent an early pathological pathway for cancer that were either overlooked by human physicians or were believed to be unimportant, according to Zhou.
That field of study is still in its infancy, though.
WARNING WORDS End-users should exercise caution when attempting to use explanation methods in practice, warns Marzyeh Ghassemi, an assistant professor and the leader of the Healthy ML Group in CSAIL. While explanation methods can occasionally be helpful for machine-learning practitioners when they are trying to catch bugs in their models or understand the inner-workings of a system.
Explanation techniques are being used to help decision makers better comprehend a model’s predictions so they will know when to trust the model and employ its guidance in practice. Machine learning has been adopted in many disciplines, from health care to education. However, Ghassemi warns against using these methods in that way .
We’ve discovered that explanations cause people—experts and nonexperts alike—to become overconfident in the wisdom of a certain recommendation system. She says, “I believe it is crucial for humans to maintain the internal circuitry that allows me to question the counsel I am given.”
She continues, citing some recent studies from Microsoft researchers, “Scientists know explanations make individuals over-confident.”
Methods of explanation are far from perfect and are not without flaws. One reason is that explanation methods have been shown by Ghassemis recent research to perpetuate biases and worse outcomes for people from disadvantaged groups.
Another drawback of explanation techniques is that it is frequently impossible to determine whether they are true in the first place. To compare the explanations to the actual model, Zhou argues, but since the user doesn’t understand how the model operates, this is circular thinking.
Zhou warns that even the best explanation should be taken with a grain of salt . He and other researchers are working to make explanation methods more faithful to the actual model predictions.
Additionally, we are prone to generalizing too much and believe that these models make decisions in a manner similar to that of humans. In order to ensure that the generalized model understanding that individuals construct from these local explanations is balanced, he continues, we need to calm people down and restrain them.
That is what Zhous most recent research aims to do.
WHERE DO MACHINE LEARNING EXPLANATION METHODS GO FROM HERE? Ghassemi contends that the research community should put more effort into understanding how information is presented to decision makers so they can grasp it. He also contends that greater regulation is required to ensure machine-learning algorithms are used properly in practice. The only solution is better explanation techniques.
Even in the business world, there seems to be a growing awareness that we cannot simply take this data, create a visually appealing dashboard, and believe that people would perform better with that. You need to see measurable gains in action, and she hopes it will result in concrete rules for how to better present information in such highly complex domains as medicine.
Zhou anticipates future research on explanation approaches for specific use cases, such as model debugging, scientific discovery, fairness audits, and safety assurance, in addition to new work aimed at improving explanations. Researchers may develop a theory that would match explanations with certain scenarios and assist avoid some of the issues associated with applying them in real-world scenarios by identifying fine-grained properties of explanation methods and the requirements of various use cases.
Thanks to AA10
Do you value the unique reporting and cleantech news coverage on CleanTechnica? Think about signing up as a patron on board game players 1 or as a CleanTechnica Member, Supporter, Technician, or Ambassador.
