COURSE UNIT TITLE

: INFORMATION AND ENTROPY

DEGREE PROGRAMMES

Description of Individual Course Units

Course Unit Code Course Unit Title Type Of Course D U L ECTS
CSC 5004 INFORMATION AND ENTROPY ELECTIVE 3 0 0 7

Offered By

Graduate School of Natural and Applied Sciences

Level of Course Unit

Second Cycle Programmes (Master's Degree)

Course Coordinator

PROFESSOR ÖZLEM EGE ORUÇ

Offered to

Computer Science
Ph.D. in Computer Science

Course Objective

The course will study how information is measured in terms of probability and entropy, and the relationships among conditional and joint entropies.
How concepts of randomness and uncertainty are related to information. Ensembles, random variables, marginal and conditional probabilities. How the metrics of information are grounded in the rules of probability.
Marginal entropy, joint entropy, conditional entropy, and the Chain Rule for entropy. Mutual information between ensembles of random variables. Why entropy is the fundamental measure of information content. At the end of the course students should be able to
1. calculate the information content of a random variable from its probability distribution
2. relate the joint, conditional, and marginal entropies of variables in terms of their coupled probabilities.

Learning Outcomes of the Course Unit

1   An understanding of fundamental ideas of information and entropy,
2   Have a good understanding of entropy in information theory.
3   An understanding of advanced structure of entropy,
4   Produce entropy and chain rule in the computer programs,
5   Develop the students analytical abilities and ability to present and criticize arguments.

Mode of Delivery

Face -to- Face

Prerequisites and Co-requisites

None

Recomended Optional Programme Components

None

Course Contents

Week Subject Description
1 1. Information Sources 1.1. Introduction 1.2. Probability Spaces and Random Variables 1.3. Random Processes and Dynamical Systems 1.4. Distributions
2 2. Entropy and Information 2.1. Introduction 2.2. Entropy and Entropy Rate 2.3. Basic Properties of Entropy 2.4. Entropy Rate
3 3.Joint Entropy and Conditional Entropy 3.1. Introduction 3.2. Joint Entropy 3.3. Conditional Entropy and Information
4 4.Relative Entropy 4.1. Introduction 4.2. Divergence 4.3. Relative Entropy
5 5. Mutual Information 5.1. Introduction 5.2. Relationship Between Entropy and Mutual Information 5.3. Chain Rules for Entropy, Relative Entropy, and Mutual Information
6 6. Entropy Rates of a Stochastic Process 6.1. Introduction 6.2. Markov Chains 6.3. Entropy Rate 6.4. Example: Entropy Rate of a Random Walk on a Weighted Graph
7 7. Data Compression 7.1. Introduction 7.2. Examples of Codes (Ex.Huffman Kodlama, Aritmetik Kodlama, Lempel Ziv Kodu, etc.)
8 Mid-term examination
9 9. Differential Entropy 9.1. Introduction 9.2. Relation of Differential Entropy to Discrete Entropy 9.3. Joint and Conditional Differential Entropy 9.4. Properties of Differential Entropy, Relative Entropy, and Mutual Information
10 10. Information Theory and Statistics 10.1. Method of Types 10.2. Law of Large Numbers 10.3. Conditional Limit Theorem 10.4. Hypothesis Testing
11 11. Maximum Entropy 11.1. Maximum Entropy Distributions 11.2. Examples
12 12. Inductive learning 12.1. Rule extraction 12.2. Examples
13 13.Applications The Entropy of English
14 Project Evaluation

Recomended or Required Reading

Textbook(s):
Cover, T. M., Thomas, J. A., Elements of Information Theory, Second Edition, John Wiley& Sons, 2006.

Supplementary Book(s):
MacKay David, J.C., Information Theory, Inference, and Learning Algorithms, U.K., 2004

Planned Learning Activities and Teaching Methods

The course consists of lecture and homework.

Assessment Methods

SORTING NUMBER SHORT CODE LONG CODE FORMULA
1 MTE MIDTERM EXAM
2 ASG ASSIGNMENT
3 FIN FINAL EXAM
4 FCG FINAL COURSE GRADE MTE * 0.40 +ASG * 0.10 +FIN * 0.50
5 RST RESIT
6 FCGR FINAL COURSE GRADE (RESIT) MTE * 0.40 +ASG * 0.10 +RST * 0.50


Further Notes About Assessment Methods

None

Assessment Criteria

Evaluation of exams, and homework.

Language of Instruction

Turkish

Course Policies and Rules

To be announced.

Contact Details for the Lecturer(s)

Assoc.Prof.Dr.Emel KURUOĞLU KANDEMIR
e-posta: emel.kuruoglu@deu.edu.tr
Tel: 0232 301 95 10

Office Hours

Will be announced.

Work Placement(s)

None

Workload Calculation

Activities Number Time (hours) Total Work Load (hours)
Lectures 14 3 42
Preparations before/after weekly lectures 12 1 12
Preparation for midterm exam 1 48 48
Preparation for final exam 1 48 48
Preparing assignments 4 7 28
Final 1 2 2
Midterm 1 2 2
TOTAL WORKLOAD (hours) 182

Contribution of Learning Outcomes to Programme Outcomes

PO/LOPO.1PO.2PO.3PO.4PO.5PO.6PO.7PO.8PO.9PO.10
LO.1433
LO.24
LO.34
LO.435
LO.54

CONTACT INFORMATION
Dokuz Eylül Üniversitesi Cumhuriyet Bulvarı No: 144 35210 Alsancak / İZMİR
Phone: +90(232) 412 12 12 - Fax: +90 (232) 464 81 35

Copyright 2015 DEÜ. BİD

HOME / ABOUT US / DEGREE PROGRAMMES / GENERAL INFORMATION FOR STUDENTS / INTERNATIONAL RELATIONS