CCSP SeminarThe online CCSP Seminar, on recent advances in communication, control and signal processing at large, is a teaching seminar. Each invited speaker is requested to present a lecture (of duration 60 - 90 minutes) that describes just one or two mathematical techniques and just as many key results. The lecture will be given at a Zoom whiteboard in classroom fashion, at classroom pace, and will be videotaped for open access if the speaker so desires.
http://ccsp.ece.umd.edu//
Universal Probability and Applications in Data Science<p>In modern statistical and data science applications, the probability distribution generating the data in question is unknown (or even absent) and decisions must be taken in a purely data-driven manner. In this talk, the information-theoretic approach of universal probability is revisited and expanded upon. This approach gives us general principles and guidelines for assigning sequential probabilities to data (based on which a decision can then be made), and has been used successfully over the years to problems in compression, prediction and estimation among others. The utility of this approach is then demonstrated through the example of universal portfolio selection with side information.</p>
<h3 id="recorded-talk">Recorded Talk</h3>
<div class="video-container">
<iframe src="https://www.youtube.com/embed/Sv7TTJFCV3k" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
</div>
Thu, 09 Nov 2023 06:21:29 -0500
http://ccsp.ece.umd.edu//2023/11/09/bhatt-universal-probability-data-science/
http://ccsp.ece.umd.edu//2023/11/09/bhatt-universal-probability-data-science/Shared Information and Markov Tree-Based Models<p>Shared information is a measure of mutual dependence among m ≥ 2 jointly distributed discrete random variables, and has been proposed as a generalization of Shannon’s mutual information. The first part of the talk will focus on some properties of shared information that make it a good measure of such mutual dependence and some applications. In the second part, I shall discuss our recent work on explicit formulae for shared information in the special case of a Markov chain on a tree and how these results help in estimating shared information when the joint distribution of the underlying random variables is not known. Joint work with Prakash Narayan.</p>
<h3 id="recorded-talk">Recorded Talk</h3>
<div class="video-container">
<iframe src="https://www.youtube.com/embed/JaCm4dOeFOw" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
</div>
Thu, 26 Oct 2023 07:21:29 -0400
http://ccsp.ece.umd.edu//2023/10/26/bhattacharya-shared-information/
http://ccsp.ece.umd.edu//2023/10/26/bhattacharya-shared-information/Sample Complexity of Distinguishing Cause from Effect<p>We study the sample complexity of causal structure learning on a two-variable system with observational and experimental data. Specifically, for two variables X and Y , we consider the classical scenario where either X causes Y , Y causes X, or there is an unmeasured confounder between X and Y . We show that if X and Y are over a finite domain of size k and are significantly correlated, the minimum number of interventional samples needed is sublinear in k. We give a tight characterization of the tradeoff between observational and interventional data when the number of observational samples is sufficiently large. We build upon techniques for closeness testing and for non-parametric density estimation in different regimes of observational data. Our hardness results are based on carefully constructing causal models whose marginal and interventional distributions form hard instances of canonical results on property testing.</p>
<h3 id="recorded-talk">Recorded Talk</h3>
<div class="video-container">
<iframe src="https://www.youtube.com/embed/sDi9Zf5w_ZU" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
</div>
Thu, 12 Oct 2023 07:21:29 -0400
http://ccsp.ece.umd.edu//2023/10/12/bhadane-distinguishing-cause-effect/
http://ccsp.ece.umd.edu//2023/10/12/bhadane-distinguishing-cause-effect/Efficient Genomic Compression using a Hierarchy of Block Codes and Novel Alignment Algorithm<p>The sequencing of genomic (DNA) data is an emerging field with vast applications in modern biological and medical research. The exploding amount of generated data requires the use of powerful compression tools. In this talk, I will present a new compression scheme for genomic data given as fragments called “reads”, which enjoys extremely low encoding complexity compared to state-of-the-art tools. The reads are assumed to be similar to segments of a reference sequence available to the decoder only, thus our multi-layered code construction carries the information needed for both aligning the (coded) read within the reference, and reconstructing the read from a similar reference segment. We first present the scheme for the case of only substitution errors between the reads and the reference, and then extend it to support reads with a single deletion and multiple substitutions. A central tool in this extension is an alignment algorithm using a new distance metric, which is shown analytically to improve alignment performance over existing distance metrics.</p>
<p>Bio:
Yuval Cassuto is an Associate Professor at the Viterbi Department of Electrical and Computer Engineering, Technion – Israel Institute of Technology. His research interests lie at the intersection of the theoretical information sciences and the engineering of practical computing and storage systems. He has served on the technical program committees of leading conferences in both theory and systems. During 2010-2011 he has been a Scientist at EPFL, the Swiss Federal Institute of Technology in Lausanne. From 2008 to 2010 he was a Research Staff Member at Hitachi Global Storage Technologies, San Jose Research Center. In 2018-2019 he held a Visiting Professor position at Western Digital Research, and a Visiting Scholar position at UC Berkeley. He received the B.Sc degree in Electrical Engineering, summa cum laude, from the Technion in 2001, and the M.S. and Ph.D. degrees in Electrical Engineering from the California Institute of Technology, in 2004 and 2008, respectively. From 2000 to 2002, he was with Qualcomm, Israel R&D Center, where he worked on modeling, design and analysis in wireless communications.
Dr. Cassuto has won the Best Student Paper Award in data storage from the IEEE Communications Society in 2010 as a student, and in 2019 as an adviser. He also won faculty awards from Qualcomm, Intel, and Western Digital. As an undergraduate student, he won the 2001 Texas Instruments DSP and Analog Challenge $100,000 prize.</p>
<h3 id="recorded-talk">Recorded Talk</h3>
<div class="video-container">
<iframe src="https://www.youtube.com/embed/O-tzmPTgFnE" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
</div>
Thu, 05 Oct 2023 07:21:29 -0400
http://ccsp.ece.umd.edu//2023/10/05/cassuto-genomic-compression-block-codes/
http://ccsp.ece.umd.edu//2023/10/05/cassuto-genomic-compression-block-codes/Smoothing of binary codes, uniform distributions, and applications<p>The action of a noise operator on a code transforms it into a distribution on the respective space. Some common examples from information theory include Bernoulli noise acting on a code in the Hamming space and Gaussian noise acting on a lattice in the Euclidean space. We aim to characterize the cases when the output distribution is close to the uniform distribution on the space, as measured by Renyi divergence of order <script type="math/tex">\alpha \in [1,\infty]</script>. A version of this question is known as the channel resolvability problem in information theory, and it has implications for security guarantees in wiretap channels, error correction, discrepancy, worst-to-average case complexity reductions, and many other problems.</p>
<p>Our work quantifies the requirements for asymptotic uniformity (perfect smoothing) and identifies explicit code families that achieve it under the action of the Bernoulli and ball noise operators on the code. We derive expressions for the minimum rate of codes required to attain asymptotically perfect smoothing. In proving our results, we leverage recent results from harmonic analysis of functions on the Hamming space. Another result pertains to the use of code families in Wyner’s transmission scheme on the binary wiretap channel. We identify explicit families that guarantee strong secrecy when applied in this scheme, showing that nested Reed-Muller codes can transmit messages reliably and securely over a binary symmetric wiretap channel with a positive rate. Finally, we establish a connection between smoothing and error correction in the binary symmetric channel.</p>
<h3 id="recorded-talk">Recorded Talk</h3>
<div class="video-container">
<iframe src="https://www.youtube.com/embed/K6VByObUkBU" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
</div>
Thu, 28 Sep 2023 07:21:29 -0400
http://ccsp.ece.umd.edu//2023/09/28/pathegama-smoothing-binary-codes/
http://ccsp.ece.umd.edu//2023/09/28/pathegama-smoothing-binary-codes/Mean Estimation in Markov Gaussian Mixture Models<p>We consider a high-dimensional mean estimation problem over a binary hidden Markov model, which illuminates the interplay between memory in data, sample size, dimension, and signal strength in statistical inference. In this model, an estimator observes n samples of a d-dimensional parameter vector <script type="math/tex">θ_∗∈ℝ^d</script>, multiplied by a random sign <script type="math/tex">S_i</script> <script type="math/tex">(1≤i≤n)</script>, and corrupted by isotropic standard Gaussian noise. The sequence of signs <script type="math/tex">\{S_i\}_{i∈[n]}∈\{−1,1\}^n</script> is drawn from a stationary homogeneous Markov chain with flip probability <script type="math/tex">δ∈[0,1/2]</script>. As <script type="math/tex">δ</script> varies, this model smoothly interpolates two well-studied models: the Gaussian Location Model for which <script type="math/tex">δ=0</script> and the Gaussian Mixture Model for which <script type="math/tex">δ=1/2</script>. Assuming that the estimator knows <script type="math/tex">δ</script>, we establish a nearly minimax optimal (up to logarithmic factors) estimation error rate, as a function of <script type="math/tex">‖θ_∗‖</script>, <script type="math/tex">δ</script>, <script type="math/tex">d</script>, <script type="math/tex">n</script>. We then provide an upper bound to the case of estimating <script type="math/tex">δ</script>, assuming a (possibly inaccurate) knowledge of <script type="math/tex">θ_∗</script>. The bound is proved to be tight when <script type="math/tex">θ_∗</script> is an accurately known constant. These results are then combined to an algorithm which estimates <script type="math/tex">θ_∗</script> with <script type="math/tex">δ</script> unknown a priori, and theoretical guarantees on its error are stated.</p>
<p>Based on joint work (arXiv:2206.02455) with Nir Weinberger.</p>
<p>Bio: Yihan Zhang received the B.Eng. degree in computer science and technology from Northeastern University, Shenyang, China, in June 2016, and the Ph.D. degree from the Department of Information Engineering, The Chinese University of Hong Kong, Hong Kong, in August 2020. He was a Post-Doctoral Researcher at the Henry and Marilyn Taub Faculty of Computer Science, Technion–Israel Institute of Technology, from October 2020 to October 2021. He has been a Post-Doctoral Researcher at the Institute of Science and Technology Austria since October 2021. His research interests include coding theory, information theory, and statistics theory (in no particular order).</p>
<h3 id="recorded-talk">Recorded Talk</h3>
<p>Coming soon!</p>
Thu, 06 Apr 2023 07:21:29 -0400
http://ccsp.ece.umd.edu//2023/04/06/zhang-mean-estimation-binary-markov/
http://ccsp.ece.umd.edu//2023/04/06/zhang-mean-estimation-binary-markov/Source coding with locality - Variations on a theme<p>The amount of data generated in many applications such as astronomy and genomics has highlighted the growing need for compression schemes that allows to interact and manipulate data directly in the compressed domain.</p>
<p>Classical compression schemes such as Lempel-Ziv are suboptimal in this regard since the recovery of even a single message symbol requires us to decompress the entire source. Is it possible to design compression schemes that are both space-optimal (achieving compressed lengths close to entropy) and yet queries on the original text can be answered efficiently by probing a small number of compressed bits?</p>
<p>I will briefly introduce variations of this problem, and much of the talk will focus on the problem of providing random access (also called local decodability or locality) in the compressed domain.</p>
<p>Rather surprisingly, lossless compression of a random source can be performed with a notion of strong locality at rates close to entropy; any individual source symbol can be decoded from a constant (independent of the length of the source sequence, n) number of compressed bits, with a vanishing in n probability of error. I will briefly discuss some known results, and our recent work in this area. I will then talk about the distributed compression setting, and show that for two separately encoded correlated sources (a.k.a. the Slepian-Wolf setup), lossless compression and strong locality is generally not achievable.</p>
<p>This is based on joint work with Aslan Tchamkerten and Venkat Chandar.</p>
<h3 id="recorded-talk">Recorded Talk</h3>
<p>Coming soon!</p>
<!--<div class="video-container">
<iframe src="https://www.youtube.com/embed/rktZx3cTDFg" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
</div>-->
Thu, 30 Mar 2023 07:21:29 -0400
http://ccsp.ece.umd.edu//2023/03/30/vatedka-source-coding/
http://ccsp.ece.umd.edu//2023/03/30/vatedka-source-coding/Independent Learning in Stochastic Games - Where Strategic Decision-Making Meets Reinforcement Learning<p>Reinforcement learning (RL) has recently achieved great successes in many sequential decision-making applications. Many of the forefront applications of RL involve the decision-making of multiple strategic agents, e.g., playing chess and Go games, autonomous driving, and robotics. Unfortunately, classical RL framework is inappropriate for multi-agent learning as it assumes an agent’s environment is stationary and does not take into account the adaptive nature of behavior. In this talk, I focus on stochastic games for multi-agent reinforcement learning in dynamic environments, and develop independent learning dynamics for stochastic games: each agent is myopic and chooses best-response type actions to other agents’ strategies independently, meaning without any coordination with her opponents. I will present our independent learning dynamics that guarantee convergence in stochastic games, including for both zero-sum and single-controller identical-interest settings. Time-permitting, I will also discuss our other results along the line of learning in stochastic games, including both the positive ones on the sample and iteration complexity of certain multi-agent RL algorithms, and negative ones on the computation complexity of general-sum stochastic games.</p>
<p>Bio: Kaiqing Zhang is currently an Assistant Professor at the Department of Electrical and Computer Engineering (ECE) and the Institute for System Research (ISR), at the University of Maryland, College Park. He is also affiliated with the Maryland Robotics Center (MRC). During the deferral time before joining Maryland, he was a postdoctoral scholar affiliated with LIDS and CSAIL at MIT, and a Research Fellow at Simons Institute for the Theory of Computing at Berkeley. He finished his Ph.D. from the Department of ECE and CSL at the University of Illinois at Urbana-Champaign (UIUC). He also received M.S. in both ECE and Applied Math from UIUC, and B.E. from Tsinghua University. His research interests lie broadly in Control and Decision Theory, Game Theory, Robotics, Reinforcement/Machine Learning, Computation, and their intersections. He is the recipient of several awards and fellowships, including Hong, McCully, and Allen Fellowship, Simons-Berkeley Research Fellowship, CSL Thesis Award, and ICML Outstanding Paper. See more details at <a href="https://kzhang66.github.io/">https://kzhang66.github.io/</a>.</p>
<h3 id="recorded-talk">Recorded Talk</h3>
<div class="video-container">
<iframe src="https://www.youtube.com/embed/rktZx3cTDFg" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
</div>
Thu, 09 Mar 2023 06:21:29 -0500
http://ccsp.ece.umd.edu//2023/03/09/zhang-independent-learning-stochastic-games/
http://ccsp.ece.umd.edu//2023/03/09/zhang-independent-learning-stochastic-games/Partial Information Decomposition in Algorithmic Fairness<p>In algorithmic fairness, when it comes to resolving legal disputes or informing policies, one needs to dig deeper and understand how the disparity arose. For instance, disparities in hiring that can be explained by an occupational necessity (code-writing for software engineering) may be exempt by law, but the disparity arising due to an aptitude test may not be (Griggs v. Duke Power). In this talk, I will discuss a question that bridges the fields of fairness, explainability, and law: how do we check if the disparity in a model is purely due to critical occupational necessities or not? We propose a systematic measure of non-exempt disparity, that brings together causality and information theory, in particular an emerging body of work in information theory called Partial Information Decomposition (PID). PID allows one to quantify the information that several random variables provide about another random variable, either individually (unique information), redundantly (shared information), or only jointly (synergistic information). To arrive at our measure of non-exempt disparity, we first examine several canonical examples that lead to a set of desirable properties (axioms) that a measure of non-exempt disparity should satisfy and then propose a measure that satisfies those properties.</p>
<p>Paper Link: <a href="https://arxiv.org/abs/2006.07986">https://arxiv.org/abs/2006.07986</a></p>
<h3 id="recorded-talk">Recorded Talk</h3>
<p>Coming soon!
<!-- <div class="video-container">
<iframe src="https://www.youtube.com/embed/wJEOwsn0I8w" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
</div> --></p>
Wed, 07 Dec 2022 06:21:29 -0500
http://ccsp.ece.umd.edu//2022/12/07/dutta-partial-information-decomposition/
http://ccsp.ece.umd.edu//2022/12/07/dutta-partial-information-decomposition/Communication complexity of two-party nonparametric estimation<p>In recent years, fundamental limits of distributed learning have
been studied under many statistical models, but often with horizontal partitioning, where data sets share the same feature space but differ in samples. Nevertheless, vertical distributed learning, where data sets differ in features, has been in use in finance and medical care. In this talk, we consider a natural distributed nonparametric estimation problem with vertically partitioned datasets. Under a given budget of communication cost or information leakage constraint, we determine the minimax rates for estimating a Holder-smooth density at a given point, which reveals that interactive protocols strictly improve upon one-way protocols. Our novel estimation scheme in the interactive setting is constructed by carefully identifying a set of auxiliary random variables. The result also implies that interactive protocols strictly improve over one-way for biased binary sequences in the Gap-Hamming problem. For global estimation of a Holder-smooth density, we characterize the minimax rates up to logarithmic factors.</p>
<p>Paper Links: <a href="https://ieeexplore.ieee.org/abstract/document/9751150">https://ieeexplore.ieee.org/abstract/document/9751150</a>, <a href="https://arxiv.org/abs/2107.00211">https://arxiv.org/abs/2107.00211</a></p>
<h3 id="recorded-talk">Recorded Talk</h3>
<p>Coming soon!
<!-- <div class="video-container">
<iframe src="https://www.youtube.com/embed/wJEOwsn0I8w" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
</div> --></p>
Tue, 22 Nov 2022 06:21:29 -0500
http://ccsp.ece.umd.edu//2022/11/22/liu-communication-complexity/
http://ccsp.ece.umd.edu//2022/11/22/liu-communication-complexity/