Abstract

Transport Layer Security (TLS) is a vital component to the security ecosystem and the most popular security protocol used on the Internet today. Despite the strengths of the protocol, numerous vulnerabilities result from its improper use in practice. Some of these vulnerabilities arise from weaknesses in authentication, from the rigidity of the trusted authority system to the complexities of client certificates. Others result from the misuse of TLS by developers, who misuse complicated TLS libraries, improperly validate server certificates, employ outdated cipher suites, or deploy other features insecurely. To make matters worse, system administrators and users are powerless to fix these issues, and lack the ability to properly control how their own machines communicate securely online.

In this dissertation we argue that the problems described are the result of an improper placement of security responsibilities. We show that by placing TLS services in the operating system, both new and existing applications can be automatically secured, developers can easily use TLS without intimate knowledge of security, and security settings can be controlled by administrators. This is demonstrated through three explorations that provide TLS features through the operating system. First, we describe and assess TrustBase, a service that repairs and strengthens certificate-based authentication for TLS connections. TrustBase uses traffic interception and a policy engine to provide administrators fine-tuned control over the trust decisions made by all applications on their systems. Second, we introduce and evaluate the Secure Socket API (SSA), which provides TLS as an operating system service through the native POSIX socket API. The SSA enables developers to use modern TLS securely, with as little as one line of code, and also allows custom tailoring of security settings by administrators. Finally, we further explore a modern approach to TLS client authentication, leveraging the operating system to provide a generic platform for strong authentication that supports easy deployment of client authentication features and protects user privacy. We conclude with a discussion of the reasons for the success of our efforts, and note avenues for future work that leverage the principles exhibited in this work, both in and beyond TLS.

Degree

PhD

College and Department

Physical and Mathematical Sciences; Computer Science

Rights

http://lib.byu.edu/about/copyright