Resolution of the dilemma in explainable Artificial Intelligence: Who is going to explain the explainer?

 Figure: Infinite
Regress (Wikipedia)

Preamble 

Surge in usage of artificial intelligence (AI) systems, now a standard practice for mid to large scale industries. These systems can not reason by construction and the legal requirements dictates if a machine learning/AI model made a decision, such as granting a loan or not for example, people affected by this decision has right to know the reason. However, it is well known that machine learning models can not reason or provide a reasoning out of box.  Apart from modifying conventional machine learning systems that includes some form of reasoning as a research exercise, practicing or building so called explainable or interpretable machine learning solutions are very popular on top of conventional models. Though there is no accepted definition of what should entail an explanation of the machine learning systems, but in general, this field of study is called explainable  artificial intelligence.

One of the most used or popularised set of techniques essentially build a secondary model on top of the primary model's behaviour and try to come up with a story on how the primary model, AI system, brought up its answers. However, this approach sounds like a good solution at the first glance, it actually trapped us into an infinite regress, a dilemma: Who is going to explain the explainer?

Avoiding 'Who is going to explain the explainer?' dilemma

Resolution of this lies in completely avoiding explainer models or techniques rely on optimisations of a similar sort. We should rely on solely so called counterfactual generators. These generators rely on a repetitive query to the system to generate data on the behaviour of the AI system to answer what if scenarios or a set of what if scenarios, corresponding to a set of reasoning statements. 

What are counterfactual generators?

Figure: Counterfactual generator,
instance based.

These are techniques that can generate a counter factual statement on the predicted machine learning decision. For example for a loan approval model, a counterfactual statement would be 'If applicants income was 10K more a model would have approved the loan". A simplest form of counterfactual generator one can think of is Individual Conditional Expectation (ICE) curves [ Goldstein2013 ], ICE curves shows, what would happen to model decision if one of the feature, such as income, vary over set of values. The idea is simple but it is so powerful that, one can generate dataset for counterfactual reasoning, so the name counterfactual generator. These are classified as model agnostic methods in general [ Du2020, Molnar ] but distinction here we are  trying to make is avoiding building another model to explain the primary model but we solely rely on queries to the model. This rules out LIME, as it relies on building models to explain the model, we question that if linear regression is intrinsically explainable here [Lipton]. One extension to ICE is generating a falling list [ wang14 ] outputs without building models.

. 

 

Outlook

We rule out of using secondary machine learning models or any models, including simple linear regression, in building an explanation for machine learning system. Instead we claim that reasoning can be achieved a simplest level with counterfactual generators based on systems behaviour to different query sets. This seems to be a good direction, as reasoning can be defined as  "algebraically manipulating previously acquired knowledge in order to answer a new question" by Léon Botton [ Botton ] and of course partly inline with Judea Pearl's causal inference revolution, though replacing the machine learning model with the causal model completely would be more causal inference recommendation.

References and further reading

[ Goldstein2013 ] Peeking Inside the Black Box: Visualising Statistical Learning with Plots of Individual Conditional Expectation, Goldstein et. al. arXiv

[ Lipton ] The Mythos of Model Interpretability, Z. Lipton arXiv

[ Molnar ] Interpretable ML book, C. Molnar url

[ Botton ] From machine learning to machine reasoning An essay, Léon Bottou doi

[ Du2020 ] Techniques for Interpretable Machine Learning, Du et. al, doi

[ wang14 ] Falling Rule Lists, Wang-Rudin arXiv

Re-discovery of Inverse problems: What is underspecification for machine learning models?

Radon, founder of 
inverse problems (Wikipedia)

This is a very well known concept in geophysics to image reconstruction communities many decades. Underspecification stems from Hadamard's definition of well-posed problem. It isn't a new problem. If you do a research on underspecification for machine learning, please do make sure that relevant literature on ill-posed problems are studied well before making strong statements. It would be helpful and prevent the reinvention of the wheel.  

One technique everyone aware of is L2 regularisation, this is to reduce ill-possedness of machine learning models. In the context of how come a deployed model's performance degrade over time, ill-possedness play a role but it isn't the sole reason. There is a large literature on inverse problems dedicated  to solve these issues, and if underspecification was the sole issue for deployed machine learning systems degrading over time: we would have reduced the performance degradation by applying strong L1-regularisations to reduce "the feature selection bias",  hence the lower the effect of underspecification. Specially in deep learning models, underspecification shouldn't be an issue, due to representation learning deep learning models bring naturally, given the inputs covers the basic learning space. 

Shannon's Entropy: Why it is called Entropy?

 

The story (legend) goes like this, von Neumann was asked by Shannon what he thinks and suggested Shannon call it entropy. 

Shannon's entropy is actually a toy version of Boltzmann's entropy. It is a toy version because it only considers configurational entropy of discrete objects without actually describing microstates. The more interesting connection where almost no-one knows is that actually, Birkhoff's ergodic theory has legitimised Shannon's entropy as his version of ergodicity is the toy version of Boltzmann. Well, Gibbs's contribution has a different angle and it is astonishing that why von Neumann omitted that is interesting.                                                                                                      

Shannon's entropy should be called von Neumann-Boltzmann-Shannon Entropy not only Shannon, maybe adding Birkhoff in the team. 

Cite as 

 @misc{suezen20sew, 

     title = {Shannon's Entropy: Why it is called Entropy? }, 

     howpublished = {\url{https://science-memo.blogspot.com/2020/11/shannons-entropy-why-it-is-called.html}, 

     author = {Mehmet Süzen},

     year = {2020}

}  

Postscripts

• Ergodicity is an intricate subject: Boltzmann's and Birkoff's differing approaches.
• Jaynes extensively studied the connection, and his interpretation was similar, he said von Neumann-Shannon expression being a " a more primitive concept" and using statistical mechanical ideas to bring in a mathematical tool for statistical inference. See his papers I and II

Can we simulate quantum state and qubits with conventional computers?

Proper simulation of quantum
computer is possible with
an other
quantum system.
Quantum Lattice NIST (Wikipedia)

The short answer is no. 

Certainly, the quantum state isn't merely a vector of complex numbers. It is inherent to the physical system which one observe quantum properties, i.e., the observer can not be removed from the observed. When we try to measure things we affect the result of the measurement. Unfortunately, contrary to popular belief, quantum systems can not be simulated with conventional computers. Numerical solutions to equations representing quantum systems are not simulations both philosophically and physically. 

Similarly, a qubit is not merely a linear combination of two complex vectors or being 1 and 0 at the same time. Qubit is also a property of a physical system and can not be simulated with a classical computer. 

Q.E.D.

Postscript Notes

• This is not new of course. Feynman has expressed the same in his landmark paper from 1982, Simulating physics with computers, here. ".. No! This is called the hidden-variable problem: it is impossible to represent the results of quantum mechanics with a classical universal device..." Richard Feynman  Feynman, R.P. Simulating physics with computers. Int J Theor Phys 21, 467–488 (1982). doi

• It is not about the hardness of simulating qubits, I.e, dynamical evolution of quantum states, the behaviour that would prove quantum advantage is a physical effect not a computational one. Entanglement is a physical process that provides computational advantage over classical computers, if it were to be replicated with numerical procedure we wouldn't have difficulty of building a quantum computer albeit a simulated one.

• Let's ask a similar question: Can we prove or conceptually show that there is a quantum advantage with simulation on the classical hardware? Unfortunately it is the same answer: No, quantum advantage can not be simulated. If we could, then we could have a simulated quantum computer on a classical hardware that solves thing much faster than the host hardware.

• Simulation is not about numerical solutions only, it means the physically intrinsic properties occurring within the simulator: This means one can simulate quantum systems or qubits only with another quantum systems, a recent example is using quantum system of ion traps to simulate another quantum system.

• Difficulty of simulating a quantum state is not about quantum dynamics : One of the hard problems in quantum computing is simulating a quantum mechanical computing device on a classical hardware. However, this is not about solving dynamics of a quantum system rather having a quantum effect on a classical system.

• A misconception in quantum computing frameworks: They don’t mean to simulate qubit as in having its behaviour replicated on a classical hardware. If you see a computational framework that claims that it can simulate a qubit, it doesn’t mean that classical hardware can replicate qubit’s behaviour, even if they solve full quantum hamiltonian dynamic evolution . Simulation in those framework implies given parameter settings and outcome is also set, one could think “simulation” in that setting as validation of already happened quantum measurement.   

• Elusive quantum state simulation : No not possible on “classical machines”
  Even one of the pioneers in quantum computing express his puzzlement of what quantum state implies, i.e., Nielsen (see What does quantum state means?).  Furthermore, current quantum computing libraries presents something called simulation mode or quantum virtual machine. Those novel works do not claim that they can simulate quantum effects on classical machine rather mimics quantum states known behaviour at the time of measurement.

• Quantum Machine Learning models can't be mapped into classical ML models
  A misconception is still repeated that we can somehow simulate or replicate artefacts of quantum computation with a classical counter part with an approximation. This is not possible due to very nature of quantum mechanical process that it can't be replicated with a classical counter part. Quantum states can't be replicated on a classical hardware, as in producing quantum advantage. 
  "..it is impossible to represent the results of quantum mechanics with a classical universal device..." Richard Feynman 
  cf.  Feynman, R.P. Simulating physics with computers. Int J Theor Phys 21, 467–488 (1982).
  More pessimistic interpretation of this statement, unfortunately, that we can't even translate data from classical hardware to quantum hardware or vice versa.

• Quantum states can not be replicated on “classical machines”: Quantum Virtual Machines (QVMs) does not claim to replicate quantum effects on classical hardware. It is a misconception to think otherwise leading to a paradox that we could have a quantum advantage on classical devices albeit a simulated one.

• Simulating quantum computers with LLMs & classical hardware 
  It doesn’t matter if we use LLMs: it isn’t possible to simulate quantum computers on classical hardware. Difficulty is not about exponential computational complexity.  Replication of effects of quantum systems on classical machines is akin to perpetual motion machine. 

• Classical hardware can’t hold qubits : Whether we use LLMs or not, it isn’t possible to simulate quantum states (computer) on a classical hardware. Difficulty is not about exponential computational complexity of quantum Hamiltonian evolution.