18-739c: Vulnerability, Defense Systems, and Malware Analysis

18-739c is a graduate course in software security in vulnerability analysis and exploitation, malware analysis, and defensive techniques. We will cover the most recent techniques found in both research as well as practice. Students will also gain hands-on experience creating exploits, analyzing malware, and developing defenses. Students will also complete a semester-long project.

18-739c is a graduate course in software security. This course is highly participatory. We will look at various proposed security measures, analyze the overall protections offered, develop new attacks, expose potential limitations, and use that knowledge to propose more secure systems.

This course has 3 main topics:

1. Vulnerability analysis: We will study how vulnerabilities are discovered and exploited. We will discuss various languages and the compilation process in order to understand what protections can be automatically applied against vulnerabilities within the constraints of the language.
2. Malware analysis: We will study current malware, as well as develop techniques for creating malware that is resilient against defensive techniques.
3. Defense analysis: We will study defenses as found in practice, and as proposed by researchers. We will also investigate the limitations of defenses.

The overall goal of this course is to introduce students to the current techniques used in both research and practice.

• Learn about latest research. We will read current and seminal research papers on vulnerability analysis, malware analysis, and system defenses. Students will prepare a short summary of each paper and submit it to the class mailing list 18739c-s11-summaries@lists.andrew.cmu.edu. Since one of the most effective ways to learn something is to try and teach it, students will prepare presentations explaining recent papers to others.
• Perform research. Students will pick a semester-long research topic of their choosing.
• Practice. This course will have two hands-on challenges. Students will:
  1. Create exploits for buffer overflows, format string vulnerabilities, and integer overflows when given only a program binary.
  2. Create malware that bypasses existing defensive techniques.

No Book

• David Brumley: By Appointment
• Sang Kil Cha: Thu. 3pm - 4pm

18-730 or permission of the instructor

We will guarantee at least the following percentages:

• 89.5-100: A
• 79.5-89.4: B
• 69.5-79.4: C
• 59.5-69.4: D
• < 59.4: F

The grade breakdown will be as follows:

• 20% Homework
• 20% In-Class Presentations
• 10% Class participation and paper summaries
• 50% Project

Exceptional work will be rewarded as appropriate.

No late days.

The course staff will treat all students ethically and fairly. We, in turn, expect the same from all students.

Any lapse in ethical behavior will immediately result in -1,000,000 points, as well as be immediately reported to the appropriate university disciplinary unit. Really. Even if you just have to pass the class, even if you didn't know it was cheating or plagiarism, and even if it will never happen again. Prof. Brumley is very, very tough and intolerant of cheating, plagiarism, or unethical behavior.

The university policy on cheating and plagiarism is available here. Note that the policy gives several examples of what constitutes cheating/plagiarism. If you have any questions, you should contact the instructor.

Please ask the course staff if you have any questions regarding whether a particular behavior is OK or not. In particular:

• Don't break laws or cause a nuisance. This course discussed security-related topics. As such, you will be exposed to ideas and techniques that could be used to break the law. This knowledge does not mean it is OK to break the law or cause a nuisance. Examples of prohibited activities include scanning networks, launching exploits, "testing" the security of a system without explicit permission from all necessary parties, and so on.
• Collaboration. Students are encouraged to talk to each other, to the course staff, or to anyone else about any of the assignments. Assistance should be limited to discussion of the problem and sketching general approaches to a solution. Each student must turn in his or her own solution.