Artificial Intelligence Stack Exchange
Artificial Intelligence Stack Exchange

Questions tagged [explainable-ai]

For questions related to explainable artificial intelligence (XAI), also known as interpretable AI, which refers to AI techniques that can be trusted and easily understood by humans, which are particularly relevant in areas like healthcare or self-driving cars. There are several concepts related to XAI, such as accountability, fairness, and transparency.

See e.g. https://en.wikipedia.org/wiki/Explainable_artificial_intelligence.

58 questions
100
votes
7 answers

Do scientists know what is happening inside artificial neural networks?

Do scientists or research experts know from the kitchen what is happening inside complex "deep" neural network with at least millions of connections firing at an instant? Do they understand the process behind this (e.g. what is happening inside and…
kenorb
  • 10,525
  • 6
  • 45
  • 95
71
votes
9 answers

Why do we need explainable AI?

If the original purpose for developing AI was to help humans in some tasks and that purpose still holds, why should we care about its explainability? For example, in deep learning, as long as the intelligence helps us to the best of their abilities…
malioboro
  • 2,859
  • 3
  • 23
  • 47
18
votes
3 answers

Which explainable artificial intelligence techniques are there?

Explainable artificial intelligence (XAI) is concerned with the development of techniques that can enhance the interpretability, accountability, and transparency of artificial intelligence and, in particular, machine learning algorithms and models,…
nbro
  • 42,615
  • 12
  • 119
  • 217
9
votes
1 answer

Why does nobody use decision trees for visual question answering?

I'm starting a project that will involve computer vision, visual question answering, and explainability. I am currently choosing what type of algorithm to use for my classifier - a neural network or a decision tree. It would seem to me that, because…
The Impossible Squish
  • 231
  • 1
  • 5
9
votes
2 answers

How is the "right to explanation" reasonable?

There has been recent uptick in interest in eXplainable Artificial Intelligence (XAI). Here is XAI's mission as stated on its DARPA page: The Explainable AI (XAI) program aims to create a suite of machine learning techniques that: Produce more…
LastIronStar
  • 91
  • 2
9
votes
1 answer

How would one debug, understand or fix the outcome of a neural network?

It seems fairly uncontroversial to say that NN based approaches are becoming quite powerful tools in many AI areas - whether recognising and decomposing images (faces at a border, street scenes in automobiles, decision making in uncertain/complex…
Stilez
  • 233
  • 1
  • 4
6
votes
0 answers

Has anyone attempted to take a bunch of similar neural networks to extract general formulae about the focus area?

When a neural network learns something from a data set, we are left with a bunch of weights which represent some approximation of knowledge about the world. Although different data sets or even different runs of the same NN might yield completely…
Lawnmower Man
  • 300
  • 1
  • 7
5
votes
2 answers

What do the neural network's weights represent conceptually?

I understand how neural networks work and have studied their theory well. My question is: On the whole, is there a clear understanding of how mutation occurs within a neural network from the input layer to the output layer, for both supervised and…
user248884
  • 151
  • 3
5
votes
1 answer

Are these visualisations the filters of the convolution layer or the convolved images with the filters?

There are several images related to convolutional networks on the Internet, an example of which I have given below My question is: are these images the weights/filters of the convolution layer (the weights that are learned in the learning process),…
Ali.kavari76
  • 121
  • 6
4
votes
1 answer

Hassija et al., "Interpreting Black-Box Models", 2024: Is this a serious paper?

"Cognitive Computation" appears to be a serious peer-reviewed Springer journal. This paper was published therein in 2024: Hassija, Vikas/Vinay Chamola/Atmesh Mahapatra/Abhinandan Singal/Divyansh Goel/Kaizhu Huang/Simone Scardapane/Indro…
Pascal
  • 43
  • 3
4
votes
1 answer

Is there any correspondence between physical objects' shape and activity occurring in a neural network?

Many years ago I saw an experiment which involved a lab primate whose brain was monitored as it watched a simple rectangular object as it moved. And sure enough, there were regions of cells in its brain that seemed, IIRC, to sequentially fire in…
Jeff
  • 143
  • 4
4
votes
1 answer

How can I interpret the way the neural network is producing an output for a given input?

I'm using a small neural network (2 hidden layers, 60 neurons apiece) for a rather complex binary classification problem. The network works well, but I'd like to know how it is using the inputs to perform the classification. Ultimately, I would like…
asheets
  • 153
  • 5
4
votes
1 answer

Is tabular Q-learning considered interpretable?

I am working on a research project in a domain where other related works have always resorted to deep Q-learning. The motivation of my research stems from the fact that the domain has an inherent structure to it, and should not require resorting to…
harshal.c
  • 141
  • 2
3
votes
2 answers

How do language models know what they don't know - and report it?

Again and again I ask myself what goes on in a pre-trained transformer-based language model (like ChatGPT9) when it comes to "know" that it cannot give an appropriate answer and either states it ("I have not enough information to answer this…
Hans-Peter Stricker
  • 931
  • 1
  • 8
  • 23
3
votes
1 answer

In GradCAM, why is activation strength considered an indicator of relevant regions?

In the GradCAM paper section 3 they implicitly propose that two things are needed to understand which areas of an input image contribute most to the output class (in a multi-label classification problem). That is: $A^k$ the final feature…
Alexander Soare
  • 1,379
  • 3
  • 12
  • 28
1
2 3 4