Re-discovery of Inverse problems: What is underspecification for machine learning models?

Radon, founder of 
inverse problems (Wikipedia)

This is a very well known concept in geophysics to image reconstruction communities many decades. Underspecification stems from Hadamard's definition of well-posed problem. It isn't a new problem. If you do a research on underspecification for machine learning, please do make sure that relevant literature on ill-posed problems are studied well before making strong statements. It would be helpful and prevent the reinvention of the wheel.  

One technique everyone aware of is L2 regularisation, this is to reduce ill-possedness of machine learning models. In the context of how come a deployed model's performance degrade over time, ill-possedness play a role but it isn't the sole reason. There is a large literature on inverse problems dedicated  to solve these issues, and if underspecification was the sole issue for deployed machine learning systems degrading over time: we would have reduced the performance degradation by applying strong L1-regularisations to reduce "the feature selection bias",  hence the lower the effect of underspecification. Specially in deep learning models, underspecification shouldn't be an issue, due to representation learning deep learning models bring naturally, given the inputs covers the basic learning space. 

Shannon's Entropy: Why it is called Entropy?

 

The story (legend) goes like this, von Neumann was asked by Shannon what he thinks and suggested Shannon call it entropy. 

Shannon's entropy is actually a toy version of Boltzmann's entropy. It is a toy version because it only considers configurational entropy of discrete objects without actually describing microstates. The more interesting connection where almost no-one knows is that actually, Birkhoff's ergodic theory has legitimised Shannon's entropy as his version of ergodicity is the toy version of Boltzmann. Well, Gibbs's contribution has a different angle and it is astonishing that why von Neumann omitted that is interesting.                                                                                                      

Shannon's entropy should be called von Neumann-Boltzmann-Shannon Entropy not only Shannon, maybe adding Birkhoff in the team. 

Cite as 

 @misc{suezen20sew, 

     title = {Shannon's Entropy: Why it is called Entropy? }, 

     howpublished = {\url{https://science-memo.blogspot.com/2020/11/shannons-entropy-why-it-is-called.html}, 

     author = {Mehmet Süzen},

     year = {2020}

}  

Postscripts

• Ergodicity is an intricate subject: Boltzmann's and Birkoff's differing approaches.
• Jaynes extensively studied the connection, and his interpretation was similar, he said von Neumann-Shannon expression being a " a more primitive concept" and using statistical mechanical ideas to bring in a mathematical tool for statistical inference. See his papers I and II

Can we simulate quantum state and qubits with conventional computers?

Proper simulation of quantum
computer is possible with
an other
quantum system.
Quantum Lattice NIST (Wikipedia)

The short answer is no. 

Certainly, the quantum state isn't merely a vector of complex numbers. It is inherent to the physical system which one observe quantum properties, i.e., the observer can not be removed from the observed. When we try to measure things we affect the result of the measurement. Unfortunately, contrary to popular belief, quantum systems can not be simulated with conventional computers. Numerical solutions to equations representing quantum systems are not simulations both philosophically and physically. 

Similarly, a qubit is not merely a linear combination of two complex vectors or being 1 and 0 at the same time. Qubit is also a property of a physical system and can not be simulated with a classical computer. 

Q.E.D.

Postscript Notes

• This is not new of course. Feynman has expressed the same in his landmark paper from 1982, Simulating physics with computers, here. ".. No! This is called the hidden-variable problem: it is impossible to represent the results of quantum mechanics with a classical universal device..." Richard Feynman  Feynman, R.P. Simulating physics with computers. Int J Theor Phys 21, 467–488 (1982). doi

• It is not about the hardness of simulating qubits, I.e, dynamical evolution of quantum states, the behaviour that would prove quantum advantage is a physical effect not a computational one. Entanglement is a physical process that provides computational advantage over classical computers, if it were to be replicated with numerical procedure we wouldn't have difficulty of building a quantum computer albeit a simulated one.

• Let's ask a similar question: Can we prove or conceptually show that there is a quantum advantage with simulation on the classical hardware? Unfortunately it is the same answer: No, quantum advantage can not be simulated. If we could, then we could have a simulated quantum computer on a classical hardware that solves thing much faster than the host hardware.

• Simulation is not about numerical solutions only, it means the physically intrinsic properties occurring within the simulator: This means one can simulate quantum systems or qubits only with another quantum systems, a recent example is using quantum system of ion traps to simulate another quantum system.

• Difficulty of simulating a quantum state is not about quantum dynamics : One of the hard problems in quantum computing is simulating a quantum mechanical computing device on a classical hardware. However, this is not about solving dynamics of a quantum system rather having a quantum effect on a classical system.

• A misconception in quantum computing frameworks: They don’t mean to simulate qubit as in having its behaviour replicated on a classical hardware. If you see a computational framework that claims that it can simulate a qubit, it doesn’t mean that classical hardware can replicate qubit’s behaviour, even if they solve full quantum hamiltonian dynamic evolution . Simulation in those framework implies given parameter settings and outcome is also set, one could think “simulation” in that setting as validation of already happened quantum measurement.   

• Elusive quantum state simulation : No not possible on “classical machines”
  Even one of the pioneers in quantum computing express his puzzlement of what quantum state implies, i.e., Nielsen (see What does quantum state means?).  Furthermore, current quantum computing libraries presents something called simulation mode or quantum virtual machine. Those novel works do not claim that they can simulate quantum effects on classical machine rather mimics quantum states known behaviour at the time of measurement.

• Quantum Machine Learning models can't be mapped into classical ML models
  A misconception is still repeated that we can somehow simulate or replicate artefacts of quantum computation with a classical counter part with an approximation. This is not possible due to very nature of quantum mechanical process that it can't be replicated with a classical counter part. Quantum states can't be replicated on a classical hardware, as in producing quantum advantage. 
  "..it is impossible to represent the results of quantum mechanics with a classical universal device..." Richard Feynman 
  cf.  Feynman, R.P. Simulating physics with computers. Int J Theor Phys 21, 467–488 (1982).
  More pessimistic interpretation of this statement, unfortunately, that we can't even translate data from classical hardware to quantum hardware or vice versa.

• Quantum states can not be replicated on “classical machines”: Quantum Virtual Machines (QVMs) does not claim to replicate quantum effects on classical hardware. It is a misconception to think otherwise leading to a paradox that we could have a quantum advantage on classical devices albeit a simulated one.

• Simulating quantum computers with LLMs & classical hardware 
  It doesn’t matter if we use LLMs: it isn’t possible to simulate quantum computers on classical hardware. Difficulty is not about exponential computational complexity.  Replication of effects of quantum systems on classical machines is akin to perpetual motion machine. 

• Classical hardware can’t hold qubits : Whether we use LLMs or not, it isn’t possible to simulate quantum states (computer) on a classical hardware. Difficulty is not about exponential computational complexity of quantum Hamiltonian evolution. 

Short Opinion: Extending General Relativity to N-dimensions is not even wrong leading to inventor's paradox

Bohr and Einstein (1926)
(Wikipedia)

It is original to have a solution to the generic case, so-called arbitrary N in mathematics, such as the N-dimensional case. This is considered as a generic solution in computer science as well, inventor's paradox.

However, such generalisation to higher-order objects may not be needed for reality. Mathematical beauty does not bring reality with it by default. An example is trying to generalise General Relativity [1]. I think this is a novel work in mathematics but it may not reflect our physical world as it is. This opinion is not new and probably the reason why many decades, community resisted against the attempts to lower the status of General Relativity as a special case of something higher dimensional object [2] that can not be tested. Einstein's theory of GR is good enough to explain our universe and supported with observations [3].

Trying to the extent any physical theory to higher dimensions may not be even wrong if it can not be observed.

[1]  A Generalization of Gravity, arXiv:1409.6757 
[2] Huggett, Nick and Vistarini, Tiziana (2014) Deriving General Relativity From String Theory.
[3]  On Experimental Tests of the General Theory of Relativity, American Journal of Physics 28, 340 (1960); https://doi.org/10.1119/1.1935800

PS: Links added on 13 April 2020