ACADEMICS
Course Details
ELE637 - Fundamentals of Information Theory
2024-2025 Fall term information
The course is not open this term
ELE637 - Fundamentals of Information Theory
| Program | Theoretýcal hours | Practical hours | Local credit | ECTS credit |
| MS | 3 | 0 | 3 | 8 |
| Obligation | : | Elective |
| Prerequisite courses | : | - |
| Concurrent courses | : | - |
| Delivery modes | : | Face-to-Face |
| Learning and teaching strategies | : | Lecture, Question and Answer, Problem Solving |
| Course objective | : | The objective of the course is to introduce ? the notion of entropy and information ? the fundamental limits of data compression ? the fundamental limits of data transmission systems. |
| Learning outcomes | : | Learn and use the main mathematical tools of information theory that quantify and relate information Learn fundamental limits for systems that store and compress data Learn fundamental methods of source coding Learn fundamental limits for systems that communicate data Utilize information theory in order to gain insight of and design any system that stores, processes, or communicates information |
| Course content | : | Introduction, review of probability, Entropy, relative entropy, mutual information, inequalities, The asymptotic equipartition property, Data compression, Channel capacity, Differential entropy, the Gaussian channel, Network information theory. |
| References | : | Elements of Information Theory, Cover and Thomas, Wiley Interscience; Gallager, "Claude E. Shannon: A Retrospective on His Life, Work, and Impact", IEEE; Trans. Inform. Theory, vol.47, no.7, Nov. 2001; Wyner, "Fundamental Limits in Information Theory", Proc. of the IEEE, vol.69, no.2,; Feb. 1981; Verdu, "Fifty Years of Shannon Theory", IEEE Trans. Inform. Theory, vol.44, no.6,; Oct. 1998 |
| Weeks | Topics |
|---|---|
| 1 | Review of probability theory, entropy |
| 2 | Relative entropy and mutual information |
| 3 | Jensen?s inequality and its consequences |
| 4 | Asymptotic equipartition property |
| 5 | Data compression and Kraft inequality |
| 6 | Optimal codes, Huffman codes |
| 7 | Shannon-Fano-Elias coding |
| 8 | Midterm Exam |
| 9 | Channel capacity examples |
| 10 | Channel coding theorem |
| 11 | Fano?s inequality and the converse to the coding theorem |
| 12 | Differential entropy |
| 13 | Gaussian channel |
| 14 | Network information theory |
| 15 | Final exam |
| 16 | Final exam |
| Course activities | Number | Percentage |
|---|---|---|
| Attendance | 0 | 0 |
| Laboratory | 0 | 0 |
| Application | 0 | 0 |
| Field activities | 0 | 0 |
| Specific practical training | 0 | 0 |
| Assignments | 1 | 5 |
| Presentation | 0 | 0 |
| Project | 0 | 0 |
| Seminar | 1 | 5 |
| Quiz | 0 | 0 |
| Midterms | 1 | 40 |
| Final exam | 1 | 50 |
| Total | 100 | |
| Percentage of semester activities contributing grade success | 50 | |
| Percentage of final exam contributing grade success | 50 | |
| Total | 100 | |
| Course activities | Number | Duration (hours) | Total workload |
|---|---|---|---|
| Course Duration | 14 | 3 | 42 |
| Laboratory | 0 | 0 | 0 |
| Application | 0 | 0 | 0 |
| Specific practical training | 0 | 0 | 0 |
| Field activities | 0 | 0 | 0 |
| Study Hours Out of Class (Preliminary work, reinforcement, etc.) | 14 | 8 | 112 |
| Presentation / Seminar Preparation | 0 | 0 | 0 |
| Project | 1 | 25 | 25 |
| Homework assignment | 1 | 5 | 5 |
| Quiz | 0 | 0 | 0 |
| Midterms (Study duration) | 0 | 0 | 0 |
| Final Exam (Study duration) | 1 | 25 | 25 |
| Total workload | 31 | 66 | 209 |
| Key learning outcomes | Contribution level | |||||
|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | ||
| 1. | Has general and detailed knowledge in certain areas of Electrical and Electronics Engineering in addition to the required fundamental knowledge. | |||||
| 2. | Solves complex engineering problems which require high level of analysis and synthesis skills using theoretical and experimental knowledge in mathematics, sciences and Electrical and Electronics Engineering. | |||||
| 3. | Follows and interprets scientific literature and uses them efficiently for the solution of engineering problems. | |||||
| 4. | Designs and runs research projects, analyzes and interprets the results. | |||||
| 5. | Designs, plans, and manages high level research projects; leads multidiciplinary projects. | |||||
| 6. | Produces novel solutions for problems. | |||||
| 7. | Can analyze and interpret complex or missing data and use this skill in multidiciplinary projects. | |||||
| 8. | Follows technological developments, improves him/herself , easily adapts to new conditions. | |||||
| 9. | Is aware of ethical, social and environmental impacts of his/her work. | |||||
| 10. | Can present his/her ideas and works in written and oral form effectively; uses English effectively. | |||||
1: Lowest, 2: Low, 3: Average, 4: High, 5: Highest