LLM-SECURITY

KO | EN

gitlite — search

#javascript #nodejs #python #api #hacktoberfest #deep-learning #python3 #rust #laravel #pytorch #react #cli

★ 23 Open GitHub ↗

Securing LLM's Against Top 10 OWASP Large Language Model Vulnerabilities 2024

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# LLM SECURITY 2024
![Securing LLM's Against Top 10  OWASP Large Language Model Vulnerabilities](https://github.com/GURPREETKAURJETHRA/LLM-SECURITY/blob/main/OWASP_TOP10_VULNERABILITY_LLMSECURITY.jpg)

Links to articles, tools, papers, books, etc. that contain useful educational materials relevant to the LLM Security project.

## Articles 📝
| Publication | Author | Date | Title and Link |
|-------------|--------|---------|-------------|
| WithSecure Labs | Benjamin Hull, Donato Capitella | 08-Apr-24 | [Domain-specific prompt injection detection with BERT classifier](https://labs.withsecure.com/publications/detecting-prompt-injection-bert-based-classifier) |
| WithSecure Labs | Donato Capitella | 21-Feb-24 | [Should you let ChatGPT control your browser?](https://labs.withsecure.com/publications/browser-agents-llm-prompt-injection) / [YouTube Video](https://www.youtube.com/watch?v=TxyjjsXT-vw) |
| Prompt Injection Explanation with video examples | Arnav Bathla | 12-Dec-23 | [Prompt Injection Explanation with video examples](https://uselayerup.com/blog/llm-threat-vector-part-2) |
| WithSecure Labs | Donato Capitella | 04-Dec-23 | [A Case Study in Prompt Injection for ReAct LLM Agents](https://labs.withsecure.com/publications/llm-agent-prompt-injection)/ [YouTube Video](https://www.youtube.com/watch?v=43qfHaKh0Xk) |
| Cyber Security Against AI Wiki | Aditya Rana | 04-Dec-23 | [Cyber Security AI Wiki](https://cyber-security.wiki/docs/machine-learning/learning/) |
| iFood Cybersec Team |Emanuel Valente | 04-Sep-23 | [Prompt Injection: Exploring, Preventing & Identifying Langchain Vulnerabilities](https://blog.ifoodsecurity.com/llm/ml/mlsec/langchain/cve/prompt/injection/2023/09/04/langchain-vulns.html) |
| PDF |Sandy Dunn | 15-Oct-23 | [AI Threat Mind Map](https://github.com/subzer0girl2/AI-Threat-Mind-Map) |
| Medium | Ken Huang | 11-Jun-23 | [LLM-Powered Applications’ Architecture Patterns and Security Controls](https://kenhuangus.medium.com/llm-powered-applications-architecture-patterns-and-security-controls-7a153c3ec9f4/) |
| Medium | Avinash Sinha | 02-Feb-23 | [AI-ChatGPT-Decision Making Ability- An Over Friendly Conversation with ChatGPT](https://medium.com/@aicyberking/an-over-friendly-conversation-with-chatgpt-hacking-openai-avinash-sinha-35c05e4e2dd5/) |
| Medium | Avinash Sinha | 06-Feb-23 | [AI-ChatGPT-Decision Making Ability- Hacking the Psychology of ChatGPT- ChatGPT Vs Siri](https://medium.com/@aicyberking/part-2-an-over-friendly-conversation-with-chatgpt-7ea21ce93114/) |
| Wired | Matt Burgess | 13-Apr-23 | [The Hacking of ChatGPT Is Just Getting Started](https://www.wired.com/story/chatgpt-jailbreak-generative-ai-hacking/) |
| The Math Company | Arjun Menon | 23-Jan-23 | [Data Poisoning and Its Impact on the AI Ecosystem](https://themathcompany.com/blog/data-poisoning-and-its-impact-on-the-ai-ecosystem) |
| IEEE Spectrum | Payal Dhar | 24-Mar-23 | [Protecting AI Models from “Data Poisoning”](https://spectrum.ieee.org/ai-cybersecurity-data-poisoning) |
| AMB Crypto | Suzuki Shillsalot | 30-Apr-23 | [Here’s how anyone can Jailbreak ChatGPT with these top 4 methods](https://ambcrypto.com/heres-how-to-jailbreak-chatgpt-with-the-top-4-methods-5/) |
| Techopedia | Kaushik Pal | 22-Apr-23 | [What is Jailbreaking in AI models like ChatGPT?](https://www.techopedia.com/what-is-jailbreaking-in-ai-models-like-chatgpt) |
| The Register | Thomas Claburn | 26-Apr-23 | [How prompt injection attacks hijack today's top-end AI – and it's tough to fix](https://www.theregister.com/2023/04/26/simon_willison_prompt_injection/) |
| Itemis | Rafael Tappe Maestro | 14-Feb-23 | [The Rise of Large Language Models ~ Part 2: Model Attacks, Exploits, and Vulnerabilities](https://blogs.itemis.com/en/model-attacks-exploits-and-vulnerabilities) |
| Hidden Layer | Eoin Wickens, Marta Janus | 23-Mar-23 | [The Dark Side of Large Language Models: Part 1](https://hiddenlayer.com/research/the-dark-side-of-large-language-models/) |
| Hidden Layer | Eoin Wickens, Marta Janus | 24-Mar-23 | [The Dark Side of Large Language Models: Part 2](https://hiddenlayer.com/research/the-dark-side-of-large-language-models-2/) |
| Embrace the Red | Johann Rehberger (wunderwuzzi) | 29-Mar-23 | [AI Injections: Direct and Indirect Prompt Injections and Their Implications](https://embracethered.com/blog/posts/2023/ai-injections-direct-and-indirect-prompt-injection-basics/) |
| Embrace the Red | Johann Rehberger (wunderwuzzi) | 15-Apr-23 | [Don't blindly trust LLM responses. Threats to chatbots](https://embracethered.com/blog/posts/2023/ai-injections-threats-context-matters/) |
| MufeedDVH | Mufeed | 9-Dec-22 | [Security in the age of LLMs](https://www.mufeedvh.com/llm-security/) |
| danielmiessler.com | Daniel Miessler | 15-May-23 | [The AI Attack Surface Map v1.0](https://danielmiessler.com/blog/the-ai-attack-surface-map-v1-0/) |
| Dark Reading | Gary McGraw | 20-Apr-23 | [Expert Insight: Dangers of Using Large Language Models Before They Are Baked](https://www.darkreading.com/vulnerabilities-threats/tech-insight-dangers-of-using-large-language-models-before-they-are-baked) |
| Honeycomb.io | Phillip Carter | 25-May-23 | [All the Hard Stuff Nobody Talks About when Building Products with LLMs](https://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-llm) |
| Wired | Matt Burgess | 25-May-23 | [The Security Hole at the Heart of ChatGPT and Bing](https://www.wired.com/story/chatgpt-prompt-injection-attack-security/) |
| BizPacReview | Terresa Monroe-Hamilton | 30-May-23 | [‘I was unaware’: NY attorney faces sanctions after using ChatGPT to write brief filled with ‘bogus’ citations](https://www.bizpacreview.com/2023/05/30/i-was-unaware-ny-attorney-faces-sanctions-after-using-chatgpt-to-write-brief-filled-with-bogus-citations-1363537/)  |
| Washington Post | Pranshu Verma | 18-May-23 | [A professor accused his class of using ChatGPT, putting diplomas in jeopardy](https://www.washingtonpost.com/technology/2023/05/18/texas-professor-threatened-fail-class-chatgpt-cheating/)  |
| Kudelski Security Research | Nathan Hamiel | 25-May-23 | [Reducing The Impact of Prompt Injection Attacks Through Design](https://research.kudelskisecurity.com/2023/05/25/reducing-the-impact-of-prompt-injection-attacks-through-design/)  |
| AI Village | GTKlondike | 7-June-23 | [Threat Modeling LLM Applications](https://aivillage.org/large%20language%20models/threat-modeling-llm/) |
| Embrace the Red | Johann Rehberger | 28-Mar-23 | [ChatGPT Plugin Exploit Explained](https://embracethered.com/blog/posts/2023/chatgpt-cross-plugin-request-forgery-and-prompt-injection./) |
| NVIDIA Developer | Will Pearce, Joseph Lucas | 14-Jun-23 | [NVIDIA AI Red Team: An Introduction](https://developer.nvidia.com/blog/nvidia-ai-red-team-an-introduction/) |
| Kanaries | Naomi Clarkson | 7-Apr-23 | [Google Bard Jailbreak](https://docs.kanaries.net/articles/google-bard-jailbreak)

## Official Guidance and Regulations
| Institution | Date | Title and Link |
|-------------|------|----------------|
|NIST|8-March-2023|[White Paper NIST AI 100-2e2023 (Draft)](https://csrc.nist.gov/publications/detail/white-paper/2023/03/08/adversarial-machine-learning-taxonomy-and-terminology/draft) |
|UK Information Commisioner's Office (ICO)|3-April-2023|[Generative AI: eight questions that developers and users need to ask](https://ico.org.uk/about-the-ico/media-centre/blog-generative-ai-eight-questions-that-developers-and-users-need-to-ask/) |
|UK National Cyber Security Centre (NCSC)|2-June-2023|[ChatGPT and large language models: what's the risk?](https://www.ncsc.gov.uk/blog-post/chatgpt-and-large-language-models-whats-the-risk) |
|UK National Cyber Security Centre (NCSC)|31 August 2022|[Principles for the security of machine learning](https://www.ncsc.gov.uk/collection/machine-learning) |
|European Parliament|31 August 2022|[EU AI Act: first regulation on artificial intelligence](https://www.europarl.europa.eu/news/en/headlines/society/20230601STO93804/eu-ai-act-first-regulation-on-artificial-intelligence) |

## Research Papers
| Publication | Author | Date | Title and Link |
|-------------|--------|---------|-------------|
| Arxiv | Samuel Gehman, et al | 24-Sep-20 | [REALTOXICITYPROMPTS: Evaluating Neural Toxic Degeneration in Language Models](https://arxiv.org/pdf/2009.11462.pdf) |
| Arxiv | Fabio Perez, Ian Ribeiro | 17-Nov-22 | [Ignore Previous Prompt: Attack Techniques For Language Models](https://arxiv.org/pdf/2211.09527.pdf) |
| Arxiv | Nicholas Carlini, et al | 14-Dec-20 | [Extracting Training Data from Large Language Models](https://arxiv.org/pdf/2012.07805.pdf) |
| NCC Group | Chris Anley | 06-Jul-22 | [Practical Attacks on Machine Learning Systems](https://research.nccgroup.com/wp-content/uploads/2022/07/practical-attacks-on-ml.pdf) |
| NCC Group | Jose Selvi | 5-Dec-22 | [Exploring Prompt Injection Attacks](https://research.nccgroup.com/2022/12/05/exploring-prompt-injection-attacks/) |
| Arxiv | Varshini Subhash | 22-Feb-2023 | [Can Large Language Models Change User Preference Adversarially?](https://arxiv.org/pdf/2302.10291.pdf) |
| ? | Jing Yang et al | 23 May 2023 | [A Systematic Literature Review of Information Security in Chatbots ](https://www.mdpi.com/2076-3417/13/11/6355) |
| Arxiv | Isaac et al | 18 April 2023 | [AI Product Security: A Primer for Developers](https://www.semanticscholar.org/reader/c72dda16d65c7fb19aad22af9d5bc03f409c9652)|
| OpenAI | OpenAI | 15-Mar-23 | [GPT-4 Technical Report](https://cdn.openai.com/papers/gpt-4.pdf) |
| Arxiv | Kai Greshake, et al | 05-May-23 | [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://arxiv.org/pdf/2302.12173.pdf) |
| Arxiv | Alexander Wan, et al | 01-May-23 | [Poisoning Language Models During Instruction Tuning](https://arxiv.org/pdf/2305.00944.pdf) |
| Arxiv | Leon Derczynski, et al | 31-Mar-23 | [Assessing Language Model Deployment with Risk Cards](https://arxiv.org/pdf/2303.18190.pdf) |
| Arxiv | Jan von der Assen, et al | 11-Mar-24 | [Asset-driven Threat Modeling for AI-based Systems](https://arxiv.org/pdf/2403.06512.pdf) |

## White Papers
| Publication | Author | Date | Title and Link |
|-------------|--------|---------|-------------|
| Deloitte | Deloitte AI Institute | 13-Mar-23 | [A new frontier in artificial intelligence - Implications of Generative AI for businesses](https://www2.deloitte.com/content/dam/Deloitte/us/Documents/deloitte-analytics/us-ai-institute-generative-artificial-intelligence.pdf) |
| Team8 | Team8 CISO Village | 18-Apr-23 | [Generative AI and ChatGPT Enterprise Risks](https://team8.vc/wp-content/uploads/2023/04/Team8-Generative-AI-and-ChatGPT-Enterprise-Risks.pdf) |
| Trail of Bits | Heidy Khlaaf | 7-Mar-23 | [Toward Comprehensive Risk Assessments and Assurance of AI-Based Systems](https://docs.google.com/viewer?url=https://raw.githubusercontent.com/trailofbits/publications/master/papers/toward_comprehensive_risk_assessments.pdf) |
| Security Implications of ChatGPT | Cloud Security Alliance (CSA) | 23-Apr-2023 | [Security Implications of ChatGPT](https://cloudsecurityalliance.org/artifacts/security-implications-of-chatgpt/) |

## Videos
| Service | Channel | Date | Title and Link |
|-------------|--------|---------|-------------|
| YouTube | LLM Chronicles | 29-Mar-24 | [Prompt Injection in LLM Browser Agents](https://www.youtube.com/watch?v=TxyjjsXT-vw&ab_channel=DonatoCapitella) |
| YouTube | Layerup | 03-Mar-24 | [GenAI Worms Explained: The Emerging Cyber Threat to LLMs](https://www.youtube.com/watch?v=ECvek8JFXG4&t=9s&ab_channel=Layerup) |
| YouTube | RALFKAIROS | 05-Feb-23 | [ChatGPT for Attack and Defense- AI Risks: Privacy, IP, Phishing, Ransomware-By Avinash Sinha](https://youtu.be/MZk4h2UuLmo?t=261) |
| YouTube | AI Explained | 25-Mar-23 | ['Governing Superintelligence' - Synthetic Pathogens, The Tree of Thoughts Paper and Self-Awareness](https://www.youtube.com/watch?v=irLn5-pTkL0) |
| YouTube | LiveOverflow | 14-Apr-23 | ['Attacking LLM - Prompt Injection'](https://youtu.be/Sv5OLj2nVAQ) |
| YouTube | LiveOverflow | 27-Apr-23 | ['Accidental LLM Backdoor - Prompt Tricks'](https://youtu.be/h74oXb4Kk8k) |
| YouTube | LiveOverflow | 11-May-23 | ['Defending LLM - Prompt Injection'](https://youtu.be/VbNPZ1n6_vY) |
| YouTube | Cloud Security Podcast | 30-May-23 | ['CAN LLMs BE ATTACKED!'](https://www.youtube.com/watch?v=Yl9qqt9C5lE&pp=ygUTY2FuIGxsbSBiZSBhdHRhY2tlZA%3D%3D) |
| YouTube | API Days | 28-Jun-23 | [Language AI Security at the API level:  Avoiding Hacks, Injections and Breaches](https://www.youtube.com/watch?v=KC9rV-8q-mQ) |

## Live Presentations 📚
| Service | Channel | Date | Title and Link |
|-------------|--------|---------|-------------|
| YouTube | API Days | 28-Jun-23 | [Securing LLM and NLP APIs: A Journey to Avoiding Data breaches, Attacks and More](https://github.com/GangGreenTemperTatum/speaking/tree/main/apidays/2023/interface) |

## CTFs and Wargames🌟

| Name | Type | Note | Link |
|-------------|--------|---------|-------------|
| SecDim | Attack and Defence | An attack and defence challenge where players should protect their chatbot secret phrase while attacking other players chatbot to exfiltrate theirs. | https://play.secdim.com/game/ai-battle |
| GPT Prompt Attack | Attack | Goal of this game is to come up with the shortest user input that tricks the system prompt into returning the secret key back to you. | https://ggpt.43z.one |
| Gandalf | Attack | Your goal is to make Gandalf reveal the secret password for each level. However, Gandalf will level up each time you guess the password, and will try harder not to give it away | https://gandalf.lakera.ai |

## Companion Projects💎
* [OWASP AI Security and Privacy Guide](https://owasp.org/www-project-ai-security-and-privacy-guide/)
* [Damn Vulnerable LLM Project](https://github.com/harishsg993010/DamnVulnerableLLMProject)
* [LLM Vulnerable Recruitment App](https://github.com/WithSecureLabs/llm-vulnerable-recruitment-app)
* [Damn Vulnerable LLM Agent](https://github.com/WithSecureLabs/damn-vulnerable-llm-agent)
* [OWASP Foundation Web Respository LLM Security](https://github.com/owasp/www-project-top-10-for-large-language-model-applications)

## Tools🔥
* [Set of tools to assess and improve LLM security](https://github.com/meta-llama/PurpleLlama)

## LLM Security Resources🎯 
![Securing LLM's Against Top 10  OWASP Large Language Model Vulnerabilities](https://github.com/GURPREETKAURJETHRA/LLM-SECURITY/blob/main/OWASP%20TOP%2010.jpg)
* [OWASP Top 10 for LLM Applications](https://llmtop10.com/)
* [OWASP Top 10 Vulnerabilities Checklist for LLM Applications](https://github.com/GURPREETKAURJETHRA/LLM-SECURITY/blob/main/OWASP%20TOP%2010%20FOR%20LLM%20APPLICATIONS.pdf)
* [LLM AI Cybersecurity & Governance Checklist](https://github.com/GURPREETKAURJETHRA/LLM-SECURITY/blob/main/LLM_AI_Security_and_Governance_Checklist-v1.1.pdf)
* [OWASP Top 10 for Large Language Model Applications](https://owasp.org/www-project-top-10-for-large-language-model-applications/)
* [OWASP Top 10 LLMs Vulnerabilities For Businesses In 2024](https://www.optiq.ai/blog-post/owasp-top-10-llms-vulnerabilities-for-businesses-in-2024)

LLM-SECURITY

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README