This dissertation presents a comprehensive study on the enhancement of unprivileged domain isolation in reduced instruction set computer (RISC)-V architectures, focusing on the design and implementation of the trust guard extension (TGX) framework. Embedded systems, essential parts of the evolving Internet of Things environment, are increasingly required to perform real-time complex tasks with limited computational resources. Robust security mechanisms are increasingly imperative as the role of these devices expands from controlling simple devices to managing complex, networked systems. However, traditional RISC-V techniques for memory isolation are unable to support the switching between a trusted execution environment without incurring a performance overhead.

This dissertation addresses the critical challenge of supporting domain isolation within RISC-V architectures. The primary challenge is to implement effective hardware-based memory protection mechanisms that operate efficiently within the constraints of embedded systems. Traditional software-based protections are reasonable, but they do not provide the necessary support for rapid memory permission changes. Furthermore, existing hardware-based solutions, such as Arm TrustZone, while effective, are not natively supported on RISC-V and have their own limitations.

A meticulous and structured approach was employed in implementing the TGX framework, beginning with a thorough analysis of the existing RISC-V architecture to identify key areas where domain isolation could be enhanced without significantly impacting performance. This analysis led to the development of two primary protection methods: Segment Level Memory Protection (SLMP) and Instruction Level Memory Protection (ILMP). SLMP extends the capabilities of Physical Memory Protection (PMP) by providing fine-grained, execution-oriented isolation. This allows for precise control over memory access based on execution segments, significantly reducing the risk of unauthorized access. ILMP complements this by offering dynamic, real-time access controls at the instruction level, adjusting memory access permissions based on the executing instructions to ensure compliance with security policies.

The TGX framework adopts a hybrid approach that combines inter-domain, execution-oriented isolation with intra-domain, instruction-level access controls. This approach leverages the strengths of existing technologies, such as MPK, while enhancing them with the unique capabilities of RISC-V. The framework ensures seamless and secure transitions between trust execution environment in user space without requiring software intervention at the privilege level.

This dissertation advances the state-of-the-art in domain isolation for RISC-V and provides a scalable and efficient solution for enhancing security in embedded systems. By employing a comprehensive methodology with a detailed evaluation of memory-protection features, their effectiveness, and hardware overhead implications, the dissertation offers significant contributions to the field of embedded system security. The research includes a practical hardware implementation evaluation and software overhead analysis, utilizing benchmarks such as Embench-iot to demonstrate the effectiveness of the proposed approach in real-world IoT environments. These findings and methodologies provide a foundation for future research directions aimed at further optimizing and expanding domain isolation technologies.