In 1948, Claude Shannon published "A Mathematical Theory of Communication", an article in two parts in the July and October issues of the Bell System Technical Journal. This work focuses on the problem of how best to encode the information a sender wants to transmit. In this fundamental work he used tools in probability theory, developed by Norbert Wiener, which were in their nascent stages of being applied to communication theory at that time. Shannon developed information entropy as a measure for the uncertainty in a message while essentially inventing the field of information theory.
The purpose of channel coding theory is to find codes which transmit quickly, contain many valid code words and can correct or at least detect many errors. While not mutually exclusive, performance in these areas is a trade off. So, different codes are optimal for different applications. The needed properties of this code mainly depend on the probability of errors happening during transmission. In a typical CD, the impairment is mainly dust or scratches.
Cryptography or cryptographic coding is the practice and study of techniques for secure communication in the presence of third parties (called adversaries). More generally, it is about constructing and analyzing protocols that block adversaries; various aspects in information security such as data confidentiality, data integrity, authentication, and non-repudiation are central to modern cryptography. Modern cryptography exists at the intersection of the disciplines of mathematics, computer science, and electrical engineering. Applications of cryptography include ATM cards, computer passwords, and electronic commerce.
Cryptography prior to the modern age was effectively synonymous with encryption, the conversion of information from a readable state to apparent nonsense. The originator of an encrypted message shared the decoding technique needed to recover the original information only with intended recipients, thereby precluding unwanted persons from doing the same. Since World War I and the advent of the computer, the methods used to carry out cryptology have become increasingly complex and its application more widespread.
Another concern of coding theory is designing codes that help synchronization. A code may be designed so that a phase shift can be easily detected and corrected and that multiple signals can be sent on the same channel.
Neural coding is a neuroscience-related field concerned with how sensory and other information is represented in the brain by networks of neurons. The main goal of studying neural coding is to characterize the relationship between the stimulus and the individual or ensemble neuronal responses and the relationship among electrical activity of the neurons in the ensemble. It is thought that neurons can encode both digital and analog information, and that neurons follow the principles of information theory and compress information, and detect and correcterrors in the signals that are sent throughout the brain and wider nervous system.
SignificanceWhy does evolution favor symmetric structures when they only represent a minute subset of all possible forms? Just as monkeys randomly typing into a computer language will preferentially produce outputs that can be generated by shorter algorithms, so the coding theorem from algorithmic information theory predicts that random mutations, when decoded by the process of development, preferentially produce phenotypes with shorter algorithmic descriptions. Since symmetric structures need less information to encode, they are much more likely to appear as potential variation. Combined with an arrival-of-the-frequent mechanism, this algorithmic bias predicts a much higher prevalence of low-complexity (high-symmetry) phenotypes than follows from natural selection alone and also explains patterns observed in protein complexes, RNA secondary structures, and a gene regulatory network.
Information theory is a mathematical framework for analyzing communication systems. This course examines its applications in linguistics, especially corpus linguistics, psycholinguistics, quantitative syntax, and typology. We study natural language as an efficient code for communication. We introduce the information-theoretic model of communication and concepts of entropy, mutual information, efficiency, robustness. Information-theoretic explanations for language universals in terms of efficient coding, including word length, word frequency distributions, and trade-offs of morphological complexity and word order fixedness. Information-theoretic models of language production and comprehension, including the principle of Uniform Information Density, expectation-based models, and noisy-channel models.
Course time will be spent on lectures, discussions, exercises, and demos. Evaluation will consist of a single 3-page paper presenting a proposed application of information theory to a linguistic problem and a proposed experiment or set of experiments to test the theory. There will be readings before each class, labeled "Discussion Material" in the schedule below.Lectures and in-class discussions will focus on the content from the discussion material.There are also recommended readings. Reading these will greatly increase the value of the course for you.
There has been a lot of fascinating work beyond the papers listed in the schedule about applying information theory to the study of human language.Here I will give a sampler of some work which I had to leave out of the main course.
Your grade will be determined by your final paper. In the final paper (3 pages double-spaced), you will be asked to elaborate on a proposed application of information theory to a linguistic problem, and to propose an experiment or set of experiments to test the theory.
J. RosenthalSome interesting problems in systems theory which are of fundamental importance in coding theory.In Proc. of the 36th IEEE Conference on Decision and Control, pages 4574-4579, San Diego, California, 1997. PDF PS
J. Rosenthal, E. V. YorkOn the relationship between algebraic systems theory and coding theory: Representations of codes.In Proc. of the 34th IEEE Conference on Decision and Control, pages 3271-3276, New Orleans, Louisiana, 1995. 2b1af7f3a8