LLMs for Security Compliance Assessment

Summary
-------
Ayesha Hafeez, the director of ML Solutions and Architecture at Arctic AI, discusses the use of LLMs (Language Model Models) for security compliance assessment. She explains the problem of manual compliance assessment and the benefits of automation using LLMs. Ayesha also provides an overview of the machine learning pipeline and the functional components involved in the solution.
Topics:
-------
Understanding the Standards
* Need to understand the security standards, specifically the NIST 853 standard
* Importance of baseline controls and control enhancements in improving compliance
Mapping User Responses
* Process of mapping user responses to control requirements
* Importance of understanding the language used in the responses
* Mapping responses to determine compliance
Challenges and Requirements
* Challenges faced in implementing the solution
* Specific requirements from the client
* Limited data access due to privacy concerns
* Challenges of interpreting convoluted legal compliance language
Machine Learning Pipeline
* Overview of the machine learning pipeline
* Functional components involved in the solution
* Role of conversational UI, middleware, and compliance reports
* Use of AWS services in the pipeline
LLM Implementation
* Use of LLMs for language understanding
* Pre-training and prompt fine-tuning techniques
* Focus on building automation to collect sufficient data for future fine-tuning
Functional Components
* Overview of the functional components of the solution
* Role of conversational UI, middleware, and compliance reports
* Interactive and efficient experience provided by the solution
Factors to Consider When Choosing an LM
* Factors to consider when choosing between an open-source LM and an LM as a service
* Importance of agency, compute and memory resources, dependency on third parties, and data residency and privacy
* Advantages and disadvantages of hosting an open-source LM internally versus using public APIs
Experimentation Framework
* Description of the experimentation framework used in the project
* Process of replicating the security questionnaire, generating a dataset, and fine-tuning prompts
* Importance of evaluation metrics in assessing performance
* Options for language models and client's preference for minimal technical investment
Q&A Session
* Ayesha's responses to questions from the audience
* Importance of combining human input and model-generated data for data augmentation
* Need to fine-tune the model to understand domain-specific taxonomies
* Importance of safeguarding the model through threat mitigation techniques
Fine Tuning and Promptuning
* Process of fine tuning and promptuning in language models
* Manual approach used for prompt tuning
* Heuristic employed to improve the model's performance
* Promptuning process, quality assessment, and feedback loop used to iterate and improve the model
Measuring Output Quality
* Method used to measure the quality of the model's output
* Use of cosine similarity and logic-based rules to ensure accuracy
* Use of Open Moderation API and data requirements for fine tuning