Ultimate physical limit of data storage: Connecting Bekenstein Bound to Landauer Principle

Preamble 

Bekenstein's Information
(Wikipedia)

Black holes are not too esoteric anymore after LIGO's success and successful imaging efforts. Their entropy behaves much different than the entropy of an ordinary matter. This leads to incredible discovery of so called holographic principle. The principle stating that we live on a projection of higher-dimensional manifestation of universe. On the other hand, Landauer made silently a discovery on energy expenditure of keeping information processing reversible. Similar bound put forward by Bekenstein for black holes made Landauer-Bekenstein bound for quantum gravity a natural avenue to study. 

What is the Bekenstein Bound?

Basically, this puts limits the size of a black hole, via its entropy bound. Essentially it states that, entropy of a black hole $S$ is bounded with the radius of the black hole $R$ and Energy $E$, other constants being 1, 

$$S \le R \dot E$$

What is Landauer Principle?

A limit on any process wants to delete 1 bit of information,  has to dissipate energy proportional to its temperature, 

$$ E \ge T ln 2$$ 

Again we made the constants to 1. 

Physical limit of data storage: 1 BekensteinBytes

Using both Bekenstein bound and Landauer principle one can compute the a physical limit for a data storage on a unit sphere,

$$S \sim T$$

If we scaled this with inverse of Planck area $\ell_{p}$; One bit of information proportional to Planck area and maximum attainable temperature $10^{32}$ K is scaled with this. The final value corresponds to $\approx 10^{100}$ bits. This again corresponds to about 10 Giga-Quetta-Quatta-Quatta bytes (1 Quetta byte is $10^{30}$ bytes). 

At this point in time, we can propose that $10^{100}$, would be to call 1 BekensteinBytes.  However with more fine-grain computations, the number may change. Our purpose here is to give a very rough idea about the scale of ultimate physical limitations of data storage in BekensteinBytes.

Bekenstein Information Conjecture : One cannot compress more than 1 BekensteinBytes on a smallest patch of space.

Outlook

An interesting connections in quantum gravity and computation, provides certain physical limitations on how much information we can be stored at a given unit space.  We called this Bekenstein Information Conjecture. 

Further reading

•  Insights into Bekenstein entropy with an intuitive mathematical definitions: A look into Thermodynamics of Black-holes (2023) 
• Bekenstein Bound (Wikipedia)
• Landauer's Principle (Wikipedia)
• On the relation between Bekenstein entropy and Landauer’s principle,  D. Song (2024) DOI
• Planck Scales (Wikipedia)
• Metric Prefixes (Wikipedia)

Cite as 

 @misc{suezen25datalimit, 

     title = {Ultimate physical limit of data storage: Connecting Bekenstein Bound  to Landauer Principle}, 

     howpublished = {\url{https://science-memo.blogspot.com/2025/04/ultimate-physical-limit-of-data-storage.html}}, 

     author = {Mehmet Süzen},

     year = {2025}

}  

Differentiating ensembles and sample spaces: Alignment between statistical mechanics and probability theory

Preamble 

Sample space is the primary concept introduced in any probability and statistics books and in papers. However, there needs to be more clarity about what constitutes a sample space in general: there is no explicit distinction between the unique event set and the replica sets. The resolution of this ambiguity lies in the concept of an ensemble.  The concept is first introduced by American theoretical physicist and engineer Gibbs in his book Elementary principle of statistical mechanics.  The primary utility of an ensemble is a mathematical construction that differentiates between samples and how they would form extended objects. 

In this direction, we provide the basics of constructing ensembles in a pedagogically accessible way from sample spaces that clears up a possible misconception. This usage of ensemble prevents the overuse of the term sample space for different things. We introduce some basic formal definitions.

    Figure: Gibbs's book
 introduced the concept of
ensemble (Wikipedia).

What Gibbs's had in mind by constructing statistical ensembles?

A statistical ensemble is a mathematical tool that connects statistical mechanics to thermodynamics. The concept lies in defining microscopic states for molecular dynamics; in statistics and probability, this corresponds to a set of events. Though these events are different at a microscopic level, they are sampled from a single thermodynamics ensemble, a representative of varying material properties or, in general, a set of independent random variables. In dynamics, micro-states samples an ensemble. This simple idea has helped Gibbs to build a mathematical formalism of statistical mechanics companion to Boltzmann's theories.

Differentiating sample space and ensemble in general

The primary confusion in probability theory on what constitutes a samples space is that there is no distinction between primitive events or events composed of primitive events. We call both sets sample space. This terminology easily overlooked in general as we concentrate on events set but not the primitive events set in solving practical problems.   

Definition: A primitive event $\mathscr{e}$ implies a logically distinct unit of experimental realisation that has not composed of any other events.

Definition: A sample space $\mathscr{S}$ is a set formed by all $N$ distinct primitive events $\mathscr{e}_{i}$.  

By this definition, regardless of how many fair coins are used or if a coin toss in a sequence for the experiment, the sample space is always ${H,T}$, because these are the most primitive distinct events a system can have, i.e., a single coin outcomes. However, the statistical ensemble can be different.  For example for two fair coins or coin toss in sequence of length two, corresponding ensemble of system size two reads ${HH, TT, HT, TH}$. Then, the definition of ensemble follows. 

Definition: An ensemble  $\mathscr{E}$ is a set of ordered set of primitive events $\mathscr{e}_{i}$. These event sets can be sampled with replacement but order matters, i.e., $ \{e_{i}, e_{j} \} \ne  \{e_{j}, e_{i} \}$, $i \ne j$.

Our two coin example's ensemble should be formally written as $\mathscr{E}=\{\{H,H\}, \{T,T\}, \{H,T\}, \{T,H\}\}$, as order matters members $HT$ and $TH$ are distinct. Obviously for a single toss ensemble and a sample space will be the same. 

Ergodicity makes the need for differentiation much more clear : Time and ensemble averaging 

The above distinction makes building time and ensemble averaging much easier. The term ensemble averaging is obvious as we know what would be the ensemble set and averaging over this set for a given observable.  Time averaging then could be achieved by curating a much larger set by resampling with replacement from the ensemble. Note that the resulting time-average value would not be unique, as one can generate many different sample sets from the ensemble. However, bear in mind that the definition of how to measure convergence to ergodic regime is not unique.

Conclusion

Even though the distinction we made sounds very obscure,  this alignment between statistical mechanics and probability theory may clarify the conception of ergodic regimes for general practitioners.

Further reading

• Practical understanding of ergodicity.
• A misconception in ergodicity: Identify ergodic regime not ergodic process

Please Cite:

 @misc{suezen22dess, 

     title = {Differentiating ensembles and sample spaces: Alignment between statistical mechanics and probability theory}, 

     howpublished = {\url{https://science-memo.blogspot.com/2022/11/ensembles-probability-theory.html}, 

     author = {Mehmet Süzen},

     year = {2022}

}  

Postscript

• If there are multiple events coming from set of primitive events, compositional outcomes considered to be ensemble not sample space. Sample space is a set that we sample from, either one or multiple times to build an ensemble. Ensemble notion within pure ML context was also noticed by late David J. C. MacKay, in his book Information Theory, Inference and Learning, Cambridge University Press (2003).

Periodic Spectral Ergodicity Accurately Predicts Deep Learning Generalisation

 Preamble 

    Dali (1931),
The Persistence of Memory (Wikipedia)

One of the new mathematical concepts arise due to understanding of deep learning is called periodic spectral ergodicity (PSE). The cascading PSE (cPSE) propagates over deep learning layers which can also be used as a complexity measure. cPSE actually can also predict the generalisation ability. In this post, we review this interesting  finding in an easy and short manner.

How periodic spectral ergodicity cascades over layers

We have reviewed spectral ergodicity in a gentle fashion earlier, here.  Only difference is that in real deep learning architectures, length of the eigenvalue spectrum, i.e., the number  of bins in the histogram, generated by weight matrices are not equal in size. To align them, we use something called periodic boundary conditions or turn the eigenvalues in a cyclic fashion, up to the maximum length spectra we have seen up to that layer. Here are the steps that give, the intuition of how to compute cascading periodic spectral ergodicity (cPSE).

1. We compute eigenvalue spectrum up to a layer $i$ and align the smaller spectrum with periodic boundary conditions, i.e., cyclic.

2. Compute spectral ergodicity at layers $i$ and $i-1$.

3. Compute the cascading PSE at layer $i$ simply with a distance metric $\Omega^{i}$  and $\Omega^{i-1}$. i.e.,  KL divergence in two directions, recall earlier tutorials.  

If we repeat this up to the last layer, cPSE measures the complexity of the deep learning architecture, both capturing structural and learning algorithm-wise, in a depth of a layer fashion. 

 Generalisation Gap and cPSE

Apart from being a complexity measure, cPSE predicts the generalisation gap given reference architecture i.e., it correlates with the performance almost perfectly. These findings are presented in the paper suzen2019 .

Conclusions and Outlook

The complexity of deep learning architectures are still an open research problem.  One of the most promising direction is to use cPSE in terms of capturing structural complexity as well. While other measures in the literature did not consider depth dependency, whereby cPSE appears to be the first one.

Reference

@article{suzen2019,
  title={Periodic Spectral Ergodicity: A Complexity Measure for Deep Neural Networks and Neural Architecture Search},
  author={S{\"u}zen, Mehmet and Cerd{\`a}, Joan J and Weber, Cornelius},
  journal={arXiv preprint arXiv:1911.07831},
  year={2019}
}

Cite this post as  Periodic Spectral Ergodicity Accurately Predicts Deep Learning Generalisation, Mehmet Süzen,  https://science-memo.blogspot.com/2021/11/periodic-spectral-ergodicity-predicts-generalisation-deep-learning.html 2021

Appendix 

Bristol v0.12.2 is now supporting in computing cPSE from list of matrices

from bristol import cPSE

import numpy as np

np.random.seed(42)

matrices = [np.random.normal(size=(64,64)) for _ in range(10)]

(d_layers, cpse) = cPSE.cpse_measure_vanilla(matrices) 

Can we simulate quantum state and qubits with conventional computers?

Proper simulation of quantum
computer is possible with
an other
quantum system.
Quantum Lattice NIST (Wikipedia)

The short answer is no. 

Certainly, the quantum state isn't merely a vector of complex numbers. It is inherent to the physical system which one observe quantum properties, i.e., the observer can not be removed from the observed. When we try to measure things we affect the result of the measurement. Unfortunately, contrary to popular belief, quantum systems can not be simulated with conventional computers. Numerical solutions to equations representing quantum systems are not simulations both philosophically and physically. 

Similarly, a qubit is not merely a linear combination of two complex vectors or being 1 and 0 at the same time. Qubit is also a property of a physical system and can not be simulated with a classical computer. 

Q.E.D.

Postscript Notes

• This is not new of course. Feynman has expressed the same in his landmark paper from 1982, Simulating physics with computers, here. ".. No! This is called the hidden-variable problem: it is impossible to represent the results of quantum mechanics with a classical universal device..." Richard Feynman  Feynman, R.P. Simulating physics with computers. Int J Theor Phys 21, 467–488 (1982). doi

• It is not about the hardness of simulating qubits, I.e, dynamical evolution of quantum states, the behaviour that would prove quantum advantage is a physical effect not a computational one. Entanglement is a physical process that provides computational advantage over classical computers, if it were to be replicated with numerical procedure we wouldn't have difficulty of building a quantum computer albeit a simulated one.

• Let's ask a similar question: Can we prove or conceptually show that there is a quantum advantage with simulation on the classical hardware? Unfortunately it is the same answer: No, quantum advantage can not be simulated. If we could, then we could have a simulated quantum computer on a classical hardware that solves thing much faster than the host hardware.

• Simulation is not about numerical solutions only, it means the physically intrinsic properties occurring within the simulator: This means one can simulate quantum systems or qubits only with another quantum systems, a recent example is using quantum system of ion traps to simulate another quantum system.

• Difficulty of simulating a quantum state is not about quantum dynamics : One of the hard problems in quantum computing is simulating a quantum mechanical computing device on a classical hardware. However, this is not about solving dynamics of a quantum system rather having a quantum effect on a classical system.

• A misconception in quantum computing frameworks: They don’t mean to simulate qubit as in having its behaviour replicated on a classical hardware. If you see a computational framework that claims that it can simulate a qubit, it doesn’t mean that classical hardware can replicate qubit’s behaviour, even if they solve full quantum hamiltonian dynamic evolution . Simulation in those framework implies given parameter settings and outcome is also set, one could think “simulation” in that setting as validation of already happened quantum measurement.   

• Elusive quantum state simulation : No not possible on “classical machines”
  Even one of the pioneers in quantum computing express his puzzlement of what quantum state implies, i.e., Nielsen (see What does quantum state means?).  Furthermore, current quantum computing libraries presents something called simulation mode or quantum virtual machine. Those novel works do not claim that they can simulate quantum effects on classical machine rather mimics quantum states known behaviour at the time of measurement.

• Quantum Machine Learning models can't be mapped into classical ML models
  A misconception is still repeated that we can somehow simulate or replicate artefacts of quantum computation with a classical counter part with an approximation. This is not possible due to very nature of quantum mechanical process that it can't be replicated with a classical counter part. Quantum states can't be replicated on a classical hardware, as in producing quantum advantage. 
  "..it is impossible to represent the results of quantum mechanics with a classical universal device..." Richard Feynman 
  cf.  Feynman, R.P. Simulating physics with computers. Int J Theor Phys 21, 467–488 (1982).
  More pessimistic interpretation of this statement, unfortunately, that we can't even translate data from classical hardware to quantum hardware or vice versa.

• Quantum states can not be replicated on “classical machines”: Quantum Virtual Machines (QVMs) does not claim to replicate quantum effects on classical hardware. It is a misconception to think otherwise leading to a paradox that we could have a quantum advantage on classical devices albeit a simulated one.

• Simulating quantum computers with LLMs & classical hardware 
  It doesn’t matter if we use LLMs: it isn’t possible to simulate quantum computers on classical hardware. Difficulty is not about exponential computational complexity.  Replication of effects of quantum systems on classical machines is akin to perpetual motion machine. 

• Classical hardware can’t hold qubits : Whether we use LLMs or not, it isn’t possible to simulate quantum states (computer) on a classical hardware. Difficulty is not about exponential computational complexity of quantum Hamiltonian evolution. 

Freeman Dyson's contribution to deep learning: Circular ensembles mimics trained deep neural networks

In memory of Professor Dyson, also see the paper Equivalence in Deep Neural Networks via Conjugate Matrix Ensembles

Preamble 

Dyson 2007 (Wikipedia) 

Freeman Dyson was a polymath scientist: theoretical physicist, mathematician and visionary thinker among others. In this post, we will briefly summarise his contribution to deep learning,  i.e., deep neural networks.  Obscure usage of his circular ensembles as a simulation tool in conjunction with the concept of ergodicity explained why deep learning systems learn in such high accuracy.

A simulation tool for deep learning: Circular (Random Matrix) Ensembles

Circular ensembles [1,2,3] developed by Dyson in 1962 for explaining quantum statistical mechanics systems as a modification of basic random matrix theory. Circular ensembles can be used in simulating deep learning architectures [4]. Basically, his three ensembles can be used to generate a "trained deep neural network". It is shown by myself with colleagues from Hamburg and Mallorca that using Dyson's ensembles generated networks, deeper they are so-called spectral ergodicity goes down [4], this is recently proved on real networks as well [5].

How to generate a simulated trained deep neural network in Python

Using Bristol python package [6] one could generate a set of weight matrices corresponding to each layer connections, i.e., weight matrices. A simple example, using Circular Unitary Ensemble (CUE), let's say we have 4 hidden layers of  64, 64, 128, 256 units. This would generate learned weight matrices of sizes 64x64, 64x128 and 128x256, One possible trained network weights can be generated: Note that we make non-square ones by simple multiplying by its transpose. 

from bristol.ensembles import circular
ce = circular()
seed_v   = 997123
W1 = ce.gue(64, set_seed=True, seed=seed_v)
W2 = ce.gue(128, set_seed=True, seed=seed_v)
W3 = ce.gue(256, set_seed=True, seed=seed_v)

These are complex matrices, one could take the arguments or use them as it is if only eigenvalues are needed.  An example of a trained network generation can be found in Zenedo. One can use any one of the circular ensembles.

Conclusion

Dyson's contributions are so bright that even his mathematical tools appear in modern deep learning research. He will be remembered many generations to come as a bright scientist and a polymath. 

References 

[1] Freeman Dyson, Journal of Mathematical Physics 3, 1199 (1962) [link]
[2] Michael Berry, New Journal of Physics 15 (2013) 013026 [link]
[3] Mehmet Süzen (2017), Summary Notebook on Circular ensembles [link]
[4] Spectral Ergodicity in Deep Learning Architectures via Surrogate Random Matrices,
Mehmet Süzen, Cornelius Weber, Joan J. Cerdà, arXiv:1704.08693 [link]
[5] Periodic Spectral Ergodicity: A Complexity Measure for Deep Neural Networks and Neural Architecture Search,
 Mehmet Süzen, Cornelius Weber, Joan J. Cerdà, arXiv:1911.07831 [link]
[6] Bristol Python package [link]

N-body wave function has no meaning but shut up and calculate: quantum bayesianism

During my graduate school years I was lucky to be assigned on reporting a technique used in computational quantum chemistry, namely resolution of identity.  It is a simple trick to approximate many integrals with a smaller one. I don't want to discuss about this trick but rather a peculiar idea, if not controversial,  I come up during preparation of that report (here). It is about the legitimacy of wave function in many electron, let's be brave and say N-body, physical systems. Argument is posed by Walter Kohn, Nobel Physicist, in his lecture (here) attributed to Van Vleck:

I begin with a provocative statement. In general the
many-electron wave function $\Psi(r_{1} , . . . ,r_{N} )$ for a system
of $N$ electrons is not a legitimate scientific concept, when
$N \ge N_{0}$ , where $N_{0} = 10^{3}$ .

I believe this argument has a profound implication in interpretation of quantum mechanics. However,  sadly, perception of quantum mechanics in many circles of scientific community is limited to Schrödinger equation and interpretations based on wave function. Specially the concept of wave function collapse. In my view these approaches does not make any sense for macroscopic systems such as, humans. Considering Kohn's statement. I can only see the relevance in quantum computing where really small physical systems are in consideration. (Also see my previous post, pointing out recent works in constructing quantum mechanics without wave function)

One prominent figure in recent times is distinguished Professor Mermin who popularised the short version of Copenhagen interpretation as : 'shut up and calculate' and developer of the ithaka interpretation . Very recently discusses about quantum Bayesianism (here). Again, using Kohn's statement we have to be careful not to extent this concept, again, to macroscopic systems like Humans' or even larger atomic systems, like measurement devices. So, replacing a quantum measurement device with a human observer is a mistake, if not a sin. Doesn't matter even if they have a fancy names like Bob or Alice. Because a measurement device big enough has no defined or meaningful wave function hence any kind of  "quantum probability",  i.e., no superposition to observed systems can be established. Probably, problem lies in the transition from microscopic to macroscopic system. There was a large effort in this direction (here).

What ever you believe and read about quantum mechanics. I can only suggest that your crap detectors must function fully all the time when you hear some one talk about quantum mechanics and its interpretation, including this post. I recommend you Neil Postman's article Bullshit and the Art of Crap-Detection.

PS: Recently, there was an interesting cartoon on xkcd saying that we can safely ignore any phrase that starts with "According to quantum mechanics...".

Quantum Mechanics without wavefunction

During my studies I was fortunate enough to write a short review on density functional theory, specifically reduction of computational burden in computing many electron integrals [pdf]. What strikes me in that short literature review a quote that is attributed to van Vleck about the legitimacy of wave function in standard quantum mechanics: ".. wave function has no legitimate scientific concept as pointed out by Van Vleck for many electrons (more than 10 electrons)..." according to Nobel lecture given by Walter Kohn, so the concept of density functionals in solving quantum mechanical problems. In similar lines, recently Israeli-american scientists has formulated the non-relativistic quantum mechanics without the need of wave function or density functions : Quantum states are represented as ensembles of real-valued quantum trajectories, obtained by extremizing an action and satisfying energy conservation [link] I think interpretation of this new results might be quite interesting, worth to think about it.

Semiclassical instanton theory

Semiclassical instanton approach to calculation of reaction rate constants in multidimensional chemical systems appeared in the recent work [link]. Similarly, the equivalence of two commonly used forms of semiclassical instanton theory are discussed else where
[link].

Quantum mechanical Szilard engine

A recent article on the analysis of Quantum Szilard Engine (QSZE) has appeared [link]. Japanese team derived an explicit analytical expression for work done by arbitrary number of molecules.