Algorithms

KO | EN

gitlite — search

#react #ai #llm #go #golang #rust #deep-learning #linux #docker #macos #awesome #vue #pytorch #ios

(★ 9)

Learn and implement algorithms: sorting, searching, graph theory, and dynamic programming. Includes code examples, explanations, and performance analysis.

Algorithms 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Algorithms
> Learn and implement algorithms: sorting, searching, graph theory, and dynamic programming. Includes code examples, explanations, and performance analysis.

# Contact With Me:

## Table of Contents
- [Websites](#websites)
- [YouTube Playlists](#youtube-playlists)
- [Online Courses](#online-courses)
- [Books](#books)
  - [Algorithms and Data Structures](#algorithms-and-data-structures)
  - [Algorithm Analysis](#algorithm-analysis)
  - [Randomized Algorithms](#randomized-algorithms)
- [Cheat Sheets](#cheat-sheets)
- [Github Libraries](#github-libraries)
- [Online Judges](#online-judges)
- [Blogs](#blogs)
- [Tools](#tools)
- [Algorithm Visualization](#algorithm-visualization)
  - [Projects](#projects)
  - [Libraries & Tools](#libraries--tools)
  - [References](#references)
  - [Examples](#examples)
- [Audit Algorithms](#audit-algorithms)
  - [Papers by Year](#papers-by-year)
  - [Related Events](#related-events)

## Websites

*Websites you should use to learn classic algorithms*

* [A Visual Guide to Graph Traversal Algorithms](https://workshape.github.io/visual-graph-algorithms/) - Interactive visualizations for learning how graph traversal algorithms work.
* [W3School](https://www.w3schools.in/data-structures-tutorial/intro/) - Data Structures tutorial.
* [CodeChef](https://www.codechef.com/roadmap/algorithms) - Learning DSA by practice on Codechef
* [Algorithm Visualizer](http://algo-visualizer.jasonpark.me/) - Dozens of animated algorithms (with code), and you can also create your own.
* [Algorithms Visualization](http://bost.ocks.org/mike/algorithms/) - A dense article on Algorithms Visualization.
* [Big-O Cheat Sheet](http://bigocheatsheet.com/) - Big-O complexities of common algorithms used in Computer Science.
* [Code-Drills](https://code-drills.com/tools/comparator) - Practice problems recommender (includes Codeforces, Codechef, and Spoj).
* [CP-Algorithms](https://cp-algorithms.com/) - Algorithms and data structures are especially popular in the field of competitive programming.
* [Data Structure Visualizations](http://www.cs.usfca.edu/~galles/visualization/Algorithms.html) - Visualize the behavior of Data Structures and play with its operations.
* [Geeks for Geeks](http://www.geeksforgeeks.org/fundamentals-of-algorithms/) - Lots and lots of well-explained and implemented algorithms.
* [Path Finding](https://qiao.github.io/PathFinding.js/visual/) - A visual representation of how algorithms such as A\*, IDA\*, Breadth-First-Search, Best-First-Search, and others describe a path between two points A and B.
* [Programiz](https://www.programiz.com/dsa) - Easy to follow tutorials on data structures and algorithms along with suitable examples.
* [Rosetta Code](http://rosettacode.org/wiki/Rosetta_Code) - A programming chrestomathy site that aims to present implementations of many algorithms and data structures in different programming languages.
* [Sorting Algorithms](http://www.sorting-algorithms.com/) - Nice and simple animations of sorting algorithms. With short codes and discussions.
* [Stoimen's web log](http://www.stoimen.com/) - Some algorithms nicely explained.
* [The Sound of Sorting](http://panthema.net/2013/sound-of-sorting/) - The Sound of Sorting - "Audibilization" and Visualization of Sorting Algorithms
* [VisuAlgo](http://visualgo.net) - Visualising data structures and algorithms through animation.
* [Wikipedia - Algorithms](https://en.wikipedia.org/wiki/List_of_algorithms) - Of course!!
* [Wikipedia - Data Structures](https://en.wikipedia.org/wiki/List_of_data_structures) - and why not ?!!
* [Learnersbucket](https://learnersbucket.com/) - Tutorials on data structures and algorithms in Javascript.
* [redblobgames](https://www.redblobgames.com/) - interactive visual explanations of math and algorithms, using motivating examples from computer games.

## YouTube Playlists

*High Quality Courses and tutorials on youtube*

* [Tim Roughgarden. Algorithms Illuminated](https://www.algorithmsilluminated.org/) - Tim Roughgarden, a pioneer of computer science, offers a companion to his book, with lecture videos and slides.
* [FreeCodeCamp - Algorithms and Data Structures Tutorial - Full Course for Beginners](https://www.youtube.com/watch?v=8hly31xKli0) - Complete beginner friendly Algorithms and Data Structures Tutorial with mindblowing animation.
* [Abdul Bari - Introduction to Algorithm](https://www.youtube.com/watch?v=0IAPZzGSbME&list=PLDN4rrl48XKpZkf03iYFl-O29szjTrs_O) - This playlist is so much informative and provide simulation with code example. As like as you are in a class.
* [Jenny's Lectures- Algorithm](https://www.youtube.com/watch?v=AT14lCXuMKI&list=PLdo5W4Nhv31bbKJzrsKfMpo_grxuLl8LU) - Another complete algorithm playlist from basic level to intermediate. Easy explanation and simulation.
* [MIT OpenCourseWare - Design and Analysis of Algorithms (6.046J)](https://www.youtube.com/playlist?list=PLUl4u3cNGP6317WaSNfmCvGym2ucw3oGp) - In-depth lectures on advanced algorithms by MIT professors.

## Online Courses

*Free and High-Quality Courses Online*

* [Algorithms: Divide and Conquer, Sorting and Searching, and Randomized Algorithms](https://www.coursera.org/learn/algorithms-divide-conquer) - The primary topics are: asymptotic ("Big-oh") notation, sorting and searching, divide and conquer, and randomized algorithms.
* [Algorithms: Graph Search, Shortest Paths, and Data Structures](https://www.coursera.org/learn/algorithms-graphs-data-structures) - The primary topics are: data structures, graph primitives, and their applications.
* [Algorithms: Greedy Algorithms, Minimum Spanning Trees, and Dynamic Programming](https://www.coursera.org/learn/algorithms-greedy) - The primary topics are: greedy algorithms and dynamic programming.
* [Algorithms: Shortest Paths Revisited, NP-Complete Problems and What To Do About Them](https://www.coursera.org/learn/algorithms-npcomplete) - The primary topics are: shortest paths, NP-completeness and what it means for the algorithm designer, and strategies for coping with computationally intractable problems.
* [Algorithms, Part I](https://www.coursera.org/learn/algorithms-part1/home/welcome) - This course covers the essential information that every serious programmer needs to know about algorithms and data structures. Part I covers elementary data structures, sorting, and searching algorithms. 
* [Algorithms, Part II](https://www.coursera.org/learn/algorithms-part2) - Part II focuses on graph- and string-processing algorithms.
* [Khan Academy Algorithms](https://www.khanacademy.org/computing/computer-science/algorithms) - Algorithm course ministered by Tomas Cormen and Devin Balkcom.
* [MIT - 6-006](https://ocw.mit.edu/courses/6-006-introduction-to-algorithms-fall-2011/video_galleries/lecture-videos/) - Well explained algorithms.
* [MIT - 6-046j](https://ocw.mit.edu/courses/6-046j-introduction-to-algorithms-sma-5503-fall-2005/video_galleries/video-lectures/) - Similar to the previous one, but with different algorithms.
* [MIT - 6-00sc](http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-00sc-introduction-to-computer-science-and-programming-spring-2011/index.htm) - An easy and well-explained introduction to algorithms.
* [MIT 18-409 - Topics in Theoretical Computer Science: An Algorithmist's Toolkit](https://ocw.mit.edu/courses/18-409-topics-in-theoretical-computer-science-an-algorithmists-toolkit-fall-2009/) - It covers a collection of geometric techniques that apply broadly in modern algorithm design.
* [Udacity Intro to Algorithms](https://www.udacity.com/course/intro-to-algorithms--cs215) - Python-based Algorithms course.
* [Algorithms in Motion](https://www.manning.com/livevideo/algorithms-in-motion) - Beginner's algorithms course with fun illustrations, based on the book Grokking Algorithms
* [Stanford CS161: Design and Analysis of Algorithms](https://www.edx.org/course/algorithm-design-and-analysis) - A comprehensive course covering fundamental algorithm design principles.

## Books

### Algorithms and Data structures
* [Algorithm Design](https://www.pearsonhighered.com/program/Kleinberg-Algorithm-Design/PGM319216.html) - Pretty straightforward.
* [Algorithms](http://algs4.cs.princeton.edu/home/) - Problems explained with Java, OO good practices, visualizations, and free online resources.
* [Algorithms and Data Structures in JavaScript](https://gum.co/dsajs) - Classical algorithms and data structures implemented and explained using JavaScript.
* [Algorithms in a Nutshell](https://www.amazon.com/Algorithms-Nutshell-In-OReilly/dp/059651624X) - by  George T. Heineman.
* [Classic Computer Science Problems in Python](https://www.manning.com/books/classic-computer-science-problems-in-python) -This great book presents dozens of coding challenges, ranging from simple tasks to clustering data using k-means.
* [Data Structures and Algorithms Made Easy](https://www.amazon.in/Data-Structures-Algorithms-Made-Easy/dp/819324527X) - A great way to implement algorithms with their specific programmable tasks.
* [Data Structures Using C](http://www.amazon.com/Data-Structures-Using-Aaron-Tenenbaum/dp/0131997467) - The basic concepts and usages of data structures.
* [Elementary Algorithms](https://github.com/liuxinyu95/AlgoXY) - An awesome book about algorithms and data structures.
* [Grokking Algorithms](http://www.manning.com/bhargava) - An illustrated book on algorithms with practical examples.
* [Introduction to Algorithms](http://mitpress.mit.edu/books/introduction-algorithms) - Essential!
* [Real World Algorithms: A Beginner's Guide](https://mitpress.mit.edu/books/real-world-algorithms) - An introduction to algorithms for readers with no background in advanced mathematics or computer science.
* [Swift Algorithms & Data Structures](http://shop.waynewbishop.com/) - A practical guide to concepts, theory, and code.
* [The Algorithm Design Manual](http://www.algorist.com/) - Easy to read and full of real-world examples.
* [The Art of Computer Programming](http://www-cs-faculty.stanford.edu/~uno/taocp.html) - The Book.
* [Structure and Interpretation of Computer Programs](https://mitpress.mit.edu/books/structure-and-interpretation-computer-programs-second-edition)
* [Advanced Algorithms and Data Structures](https://www.manning.com/books/advanced-algorithms-and-data-structures) - A different and a great way to introduce algorithms and data structures that can be used at work.
* [Algorithmic Puzzles](https://www.amazon.com/Algorithmic-Puzzles-Anany-Levitin/dp/0199740445) - A very accessible illustration of algorithms in the forms of puzzles. No programming experience is required!
* [Standford CS166](https://web.stanford.edu/class/cs166/) - Standford CS166, a course in the design, analysis, and implementation of data structures.
* [Grokking Data Structures](https://www.manning.com/books/grokking-data-structures) - Grokking Data Structures carefully guides you from the most basic data structures like arrays or linked lists all the way to powerful structures like graphs. It’s perfect for beginners.

### Algorithm Analysis
- [Sedgewick & Flajolet. An Introduction to the Analysis of Algorithm](https://www.amazon.com/Introduction-Analysis-Algorithms-Introdu-Algori_p2-ebook/dp/B00B3TB7IQ) - Am advanced complete survey, intended only for the mathematically matured reader.
- [McConnell. Analysis of Algorithms](https://www.amazon.com/Analysis-Algorithms-Jeffrey-McConnell/dp/0763707821) - A very accessible and brief book on algorithms analysis, with implemented code included.
- [Vrajitoru & Knight. Practical Analysis of Algorithms](https://www.amazon.com/Practical-Analysis-Algorithms-Undergraduate-Computer/dp/331909887X) - A very accessible and brief book on algorithms analysis, with implemented code included.

### Randomized Algorithms
- [Motwani & Raghavan. Randomized Algorithms](https://www.amazon.com/Randomized-Algorithms-Rajeev-Motwani/dp/0521474655) - A standard classic book.
- [Mitzenmacher & Upfal. Probability and Computing: Randomization and Probabilistic Techniques in Algorithms and Data Analysis](https://www.amazon.com/Probability-Computing-Randomization-Probabilistic-Techniques-ebook/dp/B06X9YBMFK) - Standard text for probability methods and their applications on randomized algorithms.

## Cheat Sheets
* [Tech Interview Cheat Sheet](https://github.com/TSiege/Tech-Interview-Cheat-Sheet)
* [Princeton DS Cheat Sheet](https://algs4.cs.princeton.edu/cheatsheet/)
* [CLRS in short](https://sinon.org/algorithms//#data-structures)
* [Rice university DS course in short](https://www.clear.rice.edu/comp160/data1.html)
* [Useful Reddit thread](https://www.reddit.com/r/learnprogramming/comments/3gpvyx/algorithms_and_data_structures_cheat_sheets/)
* [Algo Deck](https://github.com/teivah/algodeck/) - An open-source collection of +200 algorithmic cards.

## Github Libraries

*Implementations of the most classic algorithms in a wide variety of programming languages*

* C
    * [by @fragglet](https://github.com/fragglet/c-algorithms)
    * [by @TheAlgorithms](https://github.com/TheAlgorithms/C)
* [CoffeeScript](https://github.com/BrunoRB/algorithms.coffee)
* C#
    * [by @shkolovy](https://github.com/shkolovy/classic-algorithms)
    * [by @aalhour](https://github.com/aalhour/C-Sharp-Algorithms)
    * [by @justcoding121](https://github.com/justcoding121/Advanced-Algorithms)
* C++
    * [by @xtaci](https://github.com/xtaci/algorithms)
    * [by @PetarV-](https://github.com/PetarV-/Algorithms)
    * [by @faheel](https://github.com/faheel/Algos)
    * [by @sslotin](http://github.com/sslotin/algo)
* [Erlang](https://github.com/aggelgian/erlang-algorithms)
* Go
    * [by @arnauddri](https://github.com/arnauddri/algorithms)
    * [by @floyernick](https://github.com/floyernick/Data-Structures-and-Algorithms)
* Java
    * [by @jpa99](https://github.com/jpa99/Algorithms)
    * [by @phishman3579](https://github.com/phishman3579/java-algorithms-implementation)
    * [by @asmolich](https://github.com/asmolich/algorithms)
    * [by @psjava](https://github.com/psjava/psjava)
    * [by @jeandersonbc](https://github.com/jeandersonbc/algorithms-and-ds)
    * [by @pedrovgs](https://github.com/pedrovgs/Algorithms)
    * [by @Erdos-Graph-Framework](https://github.com/Erdos-Graph-Framework/Erdos)
    * [by @deepak-malik](https://github.com/deepak-malik/Data-Structures-In-Java)
    * [by @yusufcakal](https://github.com/yusufcakal/algorithms)
    * [by @FarheenB](https://github.com/FarheenB/Data-Structures-and-Algorithms)
* JavaScript
    * [by @jiayihu](https://github.com/jiayihu/pretty-algorithms)
    * [by @felipernb](https://github.com/felipernb/algorithms.js)
    * [by @nzakas](https://github.com/nzakas/computer-science-in-javascript)
    * [by @duereg](https://github.com/duereg/js-algorithms)
    * [by @mgechev](https://github.com/mgechev/javascript-algorithms)
    * [by @trekhleb](https://github.com/trekhleb/javascript-algorithms)
    * [by @ManrajGrover](https://github.com/ManrajGrover/algorithms-js)
    * [by @amejiarosario](https://github.com/amejiarosario/dsa.js)
    * [by @zonayedpca](https://github.com/zonayedpca/AlgoDS.js)
* Lua
    * [by @evandrolg](https://github.com/EvandroLG/computer_science_in_lua)
* Objective-C
    * [by @ EvgenyKarkan](https://github.com/EvgenyKarkan/EKAlgorithms)
* PHP
    * [by @TheAlgorithms](https://github.com/TheAlgorithms/PHP)
* Python
    * [by @nryoung](https://github.com/nryoung/algorithms)
    * [by @prakhar1989](https://github.com/prakhar1989/Algorithms)
    * [by @laurentluce](https://github.com/laurentluce/python-algorithms)
    * [by @nbro](https://github.com/nbro/ands)
    * [by @keon](https://github.com/keon/algorithms)
    * [by @vinta](https://github.com/vinta/fuck-coding-interviews)
* Ruby
    * [by @kanwei](https://github.com/kanwei/algorithms)
    * [by @sagivo](https://github.com/sagivo/algorithms)
    * [by @kumar91gopi](https://github.com/kumar91gopi/Algorithms-and-Data-Structures-in-Ruby)
* [Scala](https://github.com/vkostyukov/scalacaster)
* Swift
    * [by @kingreza](https://github.com/kingreza/Swift-Algorithms-Strings-)
    * [by @waynewbishop](https://github.com/waynewbishop/SwiftStructures)
    * [by @hollance](https://github.com/hollance/swift-algorithm-club)
* Language agnostic
    * [by @kennyledet](https://github.com/kennyledet/Algorithm-Implementations)
    * [by @indy256](https://github.com/indy256/codelibrary)
    * [by @sagivo](https://github.com/sagivo/algorithms)
    * [by @patmorin](https://github.com/patmorin/ods)
    * [by @btjanaka](https://github.com/btjanaka/algorithm-problems)

## Online Judges

*Online Judges to practice what you learned above*

* [A2 Online Judge](https://a2oj.com/) - Online Judge and problem archive.
* [ACM-ICPC Live Archive](https://icpcarchive.ecs.baylor.edu/) - Hundreds of problems from previous ACM-ICPC Regionals and World Finals.
* [AIZU ONLINE JUDGE](https://onlinejudge.u-aizu.ac.jp/) - A Japanese online judge from the University of Aizu, offering a wide range of algorithmic challenges.
* [Algo Muse](http://www.algomuse.appspot.com) - Research-based algorithmic problems. 
* [AtCoder](https://atcoder.jp/) - Japanese programming contest website.
* [Baekjoon Online Judge](https://www.acmicpc.net/) - Korean Online Judge. 10000+ problems. Supports 60+ languages.
* [Beecrowd](https://judge.beecrowd.com/) - The Largest Competitive Programming and Online Code Judgement Platform in Latin America.
* [CS Academy](https://csacademy.com/) - Holds online contests and IOI practice contests
* [CodeChef](https://www.codechef.com/) - More problems and monthly online contests.
* [Codeforces ](http://codeforces.com/) - The only programming contests Web 2.0 platform
* [Codefights](https://codefights.com/) - Practice programming and tackle your next tech interview
* [CodeMarshal](https://algo.codemarshal.org/) - Real-world contests online!
* [CodeWars](http://www.codewars.com/) - A website that houses support to solve algorithms in many languages in varying difficulty.
* [CoderByte](http://www.coderbyte.com/) - A decent website with algorithm challenges from beginner to advanced levels. Supports most of the popular languages like C++, python, javascript, ruby.
* [Firecode](https://www.firecode.io/) - Firecode.io uses machine learning algorithms along with curated real-world interview questions, solutions & a vibrant social community of learners to get you ready for your next coding interview.
* [Coding Blocks](https://hack.codingblocks.com/app/) - Website that has problems based on Maths, Data Structures, Various Algorithm and also conducts Coding Competition.
* [HackerEarth ](https://www.hackerearth.com/) - Practice algorithmic problems & challenges and participate in hiring challenges.
* [HackerRank](https://www.hackerrank.com/) - Featured algorithm and functional programming online judges
* [HiHoCoder](http://hihocoder.com/) - Chinese and English problem-solving practice and recruitment challenge site.
* [Infoarena](http://www.infoarena.ro/) - Romanian Online Judge. 1500+ algorithmic problems
* [Interviewbit](https://www.interviewbit.com/) - Learn, practice, and prepare for interviews.
* [Kattis](https://open.kattis.com/)- Online judge and problem archive
* [LavidaOnlineJudge](http://judge.lavida.us) - Korean Online Judge(Half English). 1300+ problems.
* [Learneroo Algorithms Tutorials](https://www.learneroo.com/subjects/8) - Learn and practice algorithms by solving challenges online.
* [LeetCode](https://leetcode.com/) - Learn algorithms and prepare for interviews.
* [Paiza](https://paiza.jp/) - A Japanese platform offering coding challenges and career preparation.
* [PKU JudgeOnline](http://poj.org/) - Chinese Online Judge.
* [ProjectEuler](https://projecteuler.net/) - Mathematical problems that can be solved using algorithms (or just a pencil, depending on how much you already know).
* [Rosalind](http://rosalind.info/problems/locations/) - A platform for learning bioinformatics and programming through problem-solving.
* [ShareCode.io ](https://sharecode.io/) - Online Judge and contest host with a lot of algorithmic problems in the archive to practice.
* [Snakify](https://snakify.org/) - An introductory Python course with 100+ algorithmic problems and a step-by-step debugger (from Russia).
* [SPOJ](http://www.spoj.com/) - More problems.
* [TopCoder](https://www.topcoder.com/) - Lots of problems and real-world/money-worthy problems in Graphic Design, Data Science, and Development.
* [Toph](https://toph.co/) - Bangladeshi Online Judge. Holds online contests regularly.
* [URI](https://www.urionlinejudge.com.br/judge/login) - Brazilian Online Judge. Not so many problems, but it's growing and it has online contests.
* [UVA](https://uva.onlinejudge.org/) - Hundreds of problems (from previous ACM-ICPC Regionals, World Finals, and others).
* [Codility](https://app.codility.com/programmers/challenges/) - Compete to land coding jobs at top companies
* [Google Kick Start](https://codingcompetitions.withgoogle.com/kickstart) - Global online coding competition hosted by Google.

## Blogs

*Awesome list of blogs, mainly for competitive programming but you can refer to these when learning a new topic/algorithm*

* [An awesome list for competitive programming!](https://codeforces.com/blog/entry/23054) - Awesome blog for all the resources and list of books and algorithms.
* [Algorithms Weekly](https://petr-mitrichev.blogspot.com/) - A good blog by Petr Mitrichev, mainly in Java.
* [Sport of Programming](https://www.hackerearth.com/practice/notes/getting-started-with-the-sport-of-programming/) - Informative blog for starting with the sport of programming.
* [Algorithms and Data Structures](http://www.allisons.org/ll/AlgDS/) - For getting a deeper knowledge of algorithms and how to think in the right direction.
* [Algorithm Tutorials by Tanuj Khattar](https://tanujkhattar.wordpress.com/) - Excellent blog by Tanuj Khattar. Covers tutorials on some interesting data structures along with example problems to solve.

## Tools

*Some tools that can help you in the learning of algorithms*

* [interactive-coding-challenges](https://github.com/donnemartin/interactive-coding-challenges) - Interactive, test-driven coding challenges (algorithms and data structures).
* [Visualgo](https://visualgo.net/en) - Visualising data structures and algorithms through animation.
* [Algorithm Visualizer](http://algo-visualizer.jasonpark.me/) - Dozens of animated algorithms (with code).

## Algorithm Visualization

This is a curated collection of projects and links about algorithm visualization.

### Projects
* [Setosa](http://setosa.io) blog (example: [Gridlock vs. Bottlenecks](http://setosa.io/blog/2014/09/02/gridlock/index.html))
[<img src='/images/setosa.png' width='800px'/>](http://setosa.io/blog/2014/09/02/gridlock/index.html)
* [Mathigon](http://world.mathigon.org/) (example: [Sieve of Eratosthenes](http://world.mathigon.org/Prime_Numbers)) and [beta site](http://beta.mathigon.org/)
[<img src='/images/mathigon.png' width='800px'/>](http://world.mathigon.org/Prime_Numbers)
* [Encinographic](http://encinographic.blogspot.com/) blog (example: [2D Wave Equation](http://encinographic.blogspot.com/2014/05/sim-class-wave-equation-in-2d.html))
[<img src='/images/encinographic.png' width='800px'/>](http://encinographic.blogspot.com/2014/05/sim-class-wave-equation-in-2d.html)
* [Emergent Mind](http://www.emergentmind.com/) blog (example: [Perceptrons](http://www.emergentmind.com/the-perceptron))
[<img src='/images/emergent_mind.png' width='800px'/>](http://www.emergentmind.com/the-perceptron)
* [The Nature of Code](http://natureofcode.com/book/) book (example: [Oscillation](http://natureofcode.com/book/chapter-3-oscillation/))
[<img src='/images/the_nature_of_code.png' width='800px'/>](http://natureofcode.com/book/chapter-3-oscillation/)
* [Game Mechanic Explorer](http://gamemechanicexplorer.com/) (example: [Homing Missiles](http://gamemechanicexplorer.com/#homingmissiles-1))
[<img src='/images/game_mechanic_explorer.png' width='800px'/>](http://gamemechanicexplorer.com/#homingmissiles-1)
* [Pixel Shaders](http://pixelshaders.com/proposal/) book proposal
[<img src='/images/pixel_shaders.png' width='800px'/>](http://pixelshaders.com/proposal/)
* [Probabilistic Models of Computation](https://probmods.org/) book (example: [Generative Models](https://probmods.org/chapters/generative-models.html))
[<img src='/images/probabilistic_models_of_computation.png' width='800px'/>](https://probmods.org/chapters/02-generative-models.html)
* [Steven Wittens](http://acko.net/) (example: [How to fold a Julia Fractal](http://acko.net/blog/how-to-fold-a-julia-fractal/) - click arrows on slideshows)
[<img src='/images/steve_witten.png' width='800px'/>](http://acko.net/blog/how-to-fold-a-julia-fractal/)
* [Red Blob Games](http://www.redblobgames.com/) (example: [2D Visibility](http://www.redblobgames.com/articles/visibility/))
[<img src='/images/red_blob_games.png' width='800px'/>](http://www.redblobgames.com/articles/visibility/)
* [Khan Academy algorithms course](https://www.khanacademy.org/computing/computer-science/algorithms) (example: [Breadth First Search](https://www.khanacademy.org/computing/computer-science/algorithms/breadth-first-search/a/the-breadth-first-search-algorithm))
[<img src='/images/khan_academy.png' width='800px'/>](https://www.khanacademy.org/computing/computer-science/algorithms/breadth-first-search/a/the-breadth-first-search-algorithm)
* [Christopher Olah](https://colah.github.io/) (example: [Neural Networks, Manifolds, and Topology](https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/))
[<img src='/images/colah.png' width='800px'/>](https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/)
* [Visualgo](https://visualgo.net/en)
[<img src='/images/visualgo.png' width='800px'/>](https://visualgo.net/en)
* [Melkman's Algorithm interactively explained](http://maxgoldste.in/melkman/)
[<img src='/images/melkman.png' width='800px'/>](http://maxgoldste.in/melkman/)
* [David Louapre - Science étonnante](https://scienceetonnante.com/) (example: [Le problème des trois corps](https://scienceetonnante.com/2012/07/01/le-probleme-des-trois-corps/)) - French blog with excellent visual explanations of complex scientific topics.

### Libraries & Tools

* [Tailspin](http://will.thimbleby.net/algorithms/doku.php): JS interpreter in JS that runs an algorithm and lets the visualization inspect the local variables. Records history to allow stepping backwards too.
* [Vamonos](http://rosulek.github.io/vamonos/): algorithm visualizations with pseudocode shown to reader and JS version underneath, plus some useful widgets to show pseudocode, call stack, data structures
* [Algomation](http://www.algomation.com/): algorithm visualization that uses JS generators to pass control between algorithm execution and browser event loop
* [Algorithm Visualizations](http://www.cs.usfca.edu/~galles/visualization/Algorithms.html)
* [LaTeX2HTML5](http://latex2html5.com/): annotate LaTeX diagrams with interactivity when exporting to HTML
* [Tangle](http://worrydream.com/Tangle/)

### References

* [Visualizing Algorithms](http://bost.ocks.org/mike/algorithms/)
* [Active Essays on the Web](http://tinlizzie.org/chalkboard/#ActiveEssaysOnTheWeb)
* [Visualizing sorting algorithms](https://corte.si/posts/code/visualisingsorting/index.html) - arguing that static visualizations are better than algorithm animation

### Examples

* [Garbage Collection](http://spin.atomicobject.com/2014/09/03/visualizing-garbage-collection-algorithms/)
* [Spatial Hashing](http://zufallsgenerator.github.io/2014/01/26/visually-comparing-algorithms/)
* [Fourier Image Filtering](http://david.li/filtering/)
* [Fourier Series Visualization](http://bl.ocks.org/jinroh/7524988)
* [The Skyline Problem](https://briangordon.github.io/2014/08/the-skyline-problem.html)
* [Animated Bézier Curves](http://www.jasondavies.com/animated-bezier/)
* [A Primer on Bezier Curves](http://pomax.github.io/bezierinfo/)
* [Sight And Light](http://ncase.me/sight-and-light/)
* [Newton-Raphson Optimization (1D)](http://bl.ocks.org/dannyko/ffe9653768cb80dfc0da)
* [Newton-Raphson Optimization (2D)](http://bl.ocks.org/dannyko/0956c361a6ce22362867)
* [Fisher-Yates Shuffle](http://bost.ocks.org/mike/shuffle/)
* [Tetris in Clojure](http://shaunlebron.github.io/t3tr0s-slides/#0)
* [HyperLogLog for big data distinct values](http://research.neustar.biz/2012/10/25/sketch-of-the-day-hyperloglog-cornerstone-of-a-big-data-infrastructure/)
* [Cave Culling Algorithm in Minecraft](http://tomcc.github.io/2014/08/31/visibility-1.html)
* [Explanation of X11 Servers](http://magcius.github.io/xplain/article/) has an X11 server running in the browser!
* [Manual for rot.js, a roguelike game library](http://ondras.github.io/rot.js/manual/#intro) makes all the examples editable and viewable in the browser
* [ConvNetJS](http://cs.stanford.edu/people/karpathy/convnetjs/) neural network examples in browser (example: [2-layer neural network](http://cs.stanford.edu/people/karpathy/convnetjs/demo/classify2d.html))
* [Immersive Math](http://immersivemath.com/ila/index.html) - A fully interactive linear algebra book.

## Audit Algorithms

A curated list of algorithms for auditing black-box algorithms.
Nowadays, many algorithms (recommendation, scoring, classification) are operated at third party providers, without users or institutions having any insights on how they operate on their data. Audit algorithms in this list thus apply to this setup, coined the "black-box" setup, where one auditor wants to get some insight on these remote algorithms.

> A user queries a remote algorithm (eg, through available APIs), to infer information about that algorithm.

### Papers by Year

#### 2025
- [CALM: Curiosity-Driven Auditing for Large Language Models](https://arxiv.org/abs/2501.02997) - (AAAI) *Auditing as a black-box optimization problem where the goal is to automatically uncover input-output pairs of the target LLMs that exhibit illegal, immoral, or unsafe behaviors.*
- [Queries, Representation & Detection: The Next 100 Model Fingerprinting Schemes](https://arxiv.org/abs/2412.13021) - (AAAI) *Divides model fingerprinting into three core components, to identify ∼100 previously unexplored combinations of these and gain insights into their performance.*

#### 2024
- [Auditing Local Explanations is Hard](https://arxiv.org/abs/2407.13281) - (NeurIPS) *Gives the (prohibitive) query complexity of auditing explanations.*
- [LLMs hallucinate graphs too: a structural perspective](https://arxiv.org/abs/2409.00159) - (complex networks) *Queries LLMs for known graphs and studies topological hallucinations. Proposes a structural hallucination rank.*
- [Fairness Auditing with Multi-Agent Collaboration](https://arxiv.org/pdf/2402.08522) - (ECAI) *Considers multiple agents working together, each auditing the same platform for different tasks.*
- [Mapping the Field of Algorithm Auditing: A Systematic Literature Review Identifying Research Trends, Linguistic and Geographical Disparities](https://arxiv.org/pdf/2401.11194) - (Arxiv) *Systematic review of algorithm auditing studies and identification of trends in their methodological approaches.*
- [FairProof: Confidential and Certifiable Fairness for Neural Networks](https://arxiv.org/pdf/2402.12572v1.pdf) - (Arxiv) *Proposes an alternative paradigm to traditional auditing using cryptographic tools like Zero-Knowledge Proofs; gives a system called FairProof for verifying fairness of small neural networks.*
- [Under manipulations, are some AI models harder to audit?](https://grodino.github.io/projects/manipulated-audits/preprint.pdf) - (SATML) *Relates the difficulty of black-box audits to the capacity of the targeted models, using the Rademacher complexity.*
- [Improved Membership Inference Attacks Against Language Classification Models](https://arxiv.org/pdf/2310.07219.pdf) - (ICLR) *Presents a framework for running membership inference attacks against classifier, in audit mode.*
- [Auditing Fairness by Betting](https://arxiv.org/pdf/2305.17570.pdf) - (Neurips) [[Code]](https://github.com/bchugg/auditing-fairness) *Sequential methods that allows for the continuous monitoring of incoming data from a black-box classifier or regressor.*

#### 2023
- [Privacy Auditing with One (1) Training Run](https://neurips.cc/virtual/2023/poster/70925) - (NeurIPS - best paper) *A scheme for auditing differentially private machine learning systems with a single training run.*
- [Auditing fairness under unawareness through counterfactual reasoning](https://www.sciencedirect.com/science/article/pii/S0306457322003259) - (Information Processing & Management) *Shows how to unveil whether a black-box model, complying with the regulations, is still biased or not.*
- [XAudit : A Theoretical Look at Auditing with Explanations](https://arxiv.org/pdf/2206.04740.pdf) - (Arxiv) *Formalizes the role of explanations in auditing and investigates if and how model explanations can help audits.*
- [Keeping Up with the Language Models: Robustness-Bias Interplay in NLI Data and Models](https://arxiv.org/pdf/2305.12620.pdf) - (Arxiv) *Proposes a way to extend the shelf-life of auditing datasets by using language models themselves; also finds problems with the current bias auditing metrics and proposes alternatives -- these alternatives highlight that model brittleness superficially increased the previous bias scores.*
- [Online Fairness Auditing through Iterative Refinement](https://dl.acm.org/doi/pdf/10.1145/3580305.3599454) - (KDD) *Provides an adaptive process that automates the inference of probabilistic guarantees associated with estimating fairness metrics.*
- [Stealing the Decoding Algorithms of Language Models](https://people.cs.umass.edu/~amir/papers/CCS23-LM-stealing.pdf) - (CCS) *Steal the type and hyperparameters of the decoding algorithms of a LLM.*
- [Modeling rabbit‑holes on YouTube](https://link.springer.com/epdf/10.1007/s13278-023-01105-9?sharing_token=h-O-asHI49VUWS9FxN1Gsve4RwlQNchNByi7wbcMAY6I98PKW1PqhFQJ_JqQyk3TrB05qDb3LUzMDmKOgrupccQliViDle-rwKEi2MZ8xBViaAQhyN41oZBKLLeXchoeIW2kklVHC094I5KD8pxja4-if6-iB0uAI1FnqnYoxjU%3D) - (SNAM) *Models the trapping dynamics of users in rabbit holes in YouTube, and provides a measure of this enclosure.*
- [Auditing YouTube’s Recommendation Algorithm for Misinformation Filter Bubbles](https://dl.acm.org/doi/full/10.1145/3568392) - (Transactions on Recommender Systems) *What it takes to “burst the bubble,” i.e., revert the bubble enclosure from recommendations.*
- [Auditing Yelp’s Business Ranking and Review Recommendation Through the Lens of Fairness](https://arxiv.org/pdf/2308.02129.pdf) - (Arxiv) *Audits the fairness of Yelp’s business ranking and review recommendation systems, with demographic parity, exposure, and statistical tests such as quantile linear and logistic regression.*
- [Confidential-PROFITT: Confidential PROof of FaIr Training of Trees](https://openreview.net/pdf?id=iIfDQVyuFD) - (ICLR) *Proposes fair decision tree learning algorithms along with zero-knowledge proof protocols to obtain a proof of fairness on the audited server.*
- [SCALE-UP: An Efficient Black-box Input-level Backdoor Detection via Analyzing Scaled Prediction Consistency](https://arxiv.org/pdf/2302.03251.pdf) - (ICLR) *Considers backdoor detection under the black-box setting in machine learning as a service (MLaaS) applications.*

#### 2022
- [Two-Face: Adversarial Audit of Commercial Face Recognition Systems](https://ojs.aaai.org/index.php/ICWSM/article/view/19300/19072) - (ICWSM) *Performs an adversarial audit on multiple systems APIs and datasets, making a number of concerning observations.*
- [Scaling up search engine audits: Practical insights for algorithm auditing](https://journals.sagepub.com/doi/10.1177/01655515221093029) - (Journal of Information Science) [(Code)](https://github.com/gesiscss/WebBot) *Audits multiple search engines using simulated browsing behavior with virtual agents.*
- [A zest of lime: towards architecture-independent model distances](https://openreview.net/pdf?id=OUz_9TiTv9j) - (ICLR) *Measures the distance between two remote models using LIME.*
- [Active Fairness Auditing](https://proceedings.mlr.press/v162/yan22c/yan22c.pdf) - (ICML) *Studies of query-based auditing algorithms that can estimate the demographic parity of ML models in a query-efficient manner.*
- [Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysis](https://proceedings.neurips.cc/paper/2021/file/da94cbeff56cfda50785df477941308b-Paper.pdf) - (NeurIPS) *Sobol indices provide an efficient way to capture higher-order interactions between image regions and their contributions to a (black box) neural network’s prediction through the lens of variance.*
- [Your Echos are Heard: Tracking, Profiling, and Ad Targeting in the Amazon Smart Speaker Ecosystem](https://arxiv.org/pdf/2204.10920.pdf) - (arxiv) *Infers a link between the Amazon Echo system and the ad targeting algorithm.*

#### 2021
- [When the Umpire is also a Player: Bias in Private Label Product Recommendations on E-commerce Marketplaces](https://arxiv.org/pdf/2102.00141.pdf) - (FAccT) *Do Amazon private label products get an unfair share of recommendations and are therefore advantaged compared to 3rd party products?*
- [Everyday Algorithm Auditing: Understanding the Power of Everyday Users in Surfacing Harmful Algorithmic Behaviors](https://arxiv.org/pdf/2105.02980.pdf) - (CHI) *Makes the case for "everyday algorithmic auditing" by users.*
- [Auditing Black-Box Prediction Models for Data Minimization Compliance](https://www.cs.bu.edu/faculty/crovella/paper-archive/minimization-audit-Neurips21.pdf) - (NeurIPS) *Measures the level of data minimization satisfied by the prediction model using a limited number of queries.*
- [Setting the Record Straighter on Shadow Banning](https://arxiv.org/abs/2012.05101) - (INFOCOM)  [(Code)](https://gitlab.enseeiht.fr/bmorgan/infocom-2021) *Considers the possibility of shadow banning in Twitter (ie, the moderation black-box algorithm), and measures the probability of several hypothesis.*
- [Extracting Training Data from Large Language Models](https://arxiv.org/pdf/2012.07805.pdf) - (USENIX Security) *Extract verbatim text sequences from the GPT-2 model’s training data.*
- [FairLens: Auditing black-box clinical decision support systems](https://www.sciencedirect.com/science/article/pii/S030645732100145X?casa_token=oyjFKij269MAAAAA:w_ohScpMPNMnkDdzBqAIod5QfBgQlq5Ht9mMRSOydZpOgNG-i1yuqEmBjWN__38gOGmjNL7dVT0) - (Information Processing & Management) *Presents a pipeline to detect and explain potential fairness issues in Clinical DSS, by comparing different multi-label classification disparity measures.*
- [Auditing Algorithmic Bias on Twitter](https://dl.acm.org/doi/abs/10.1145/3447535.3462491) - (WebSci).
- [Bayesian Algorithm Execution: Estimating Computable Properties of Black-box Functions Using Mutual Information](https://proceedings.mlr.press/v139/neiswanger21a.html) - (ICML) *A budget constrained and Bayesian optimization procedure to extract properties out of a black-box algorithm.*

#### 2020
- [Black-Box Ripper: Copying black-box models using generative evolutionary algorithms](https://proceedings.neurips.cc/paper/2020/file/e8d66338fab3727e34a9179ed8804f64-Paper.pdf) - (NeurIPS) *Replicates the functionality of a black-box neural model, yet with no limit on the amount of queries (via a teacher/student scheme and an evolutionary search).*
- [Auditing radicalization pathways on YouTube](https://dl.acm.org/doi/pdf/10.1145/3351095.3372879) - (FAT*) *Studies the reachability of radical channels from each others, using random walks on static channel recommendations.*
- [Adversarial Model Extraction on Graph Neural Networks](https://arxiv.org/abs/1912.07721) - (AAAI Workshop on Deep Learning on Graphs: Methodologies and Applications) *Introduces GNN model extraction and presents a preliminary approach for this.*
- [Remote Explainability faces the bouncer problem](https://rdcu.be/b6qB4) - (Nature Machine Intelligence volume 2, pages529–539)  [(Code)](https://github.com/erwanlemerrer/bouncer_problem) *Shows the impossibility (with one request) or the difficulty to spot lies on the explanations of a remote AI decision.*
- [GeoDA: a geometric framework for black-box adversarial attacks](https://openaccess.thecvf.com/content_CVPR_2020/papers/Rahmati_GeoDA_A_Geometric_Framework_for_Black-Box_Adversarial_Attacks_CVPR_2020_paper.pdf) - (CVPR)  [(Code)](https://github.com/thisisalirah/GeoDA) *Crafts adversarial examples to fool models, in a pure blackbox setup (no gradients, inferred class only).*
- [The Imitation Game: Algorithm Selectionby Exploiting Black-Box Recommender](https://github.com/erwanlemerrer/erwanlemerrer.github.io/raw/master/files/imitation_blackbox_recommenders_netys-2020.pdf) - (Netys)  [(Code)](https://github.com/gdamaskinos/RecRank) *Parametrize a local recommendation algorithm by imitating the decision of a remote and better trained one.*
- [Auditing News Curation Systems:A Case Study Examining Algorithmic and Editorial Logic in Apple News](https://ojs.aaai.org/index.php/ICWSM/article/view/7277) - (ICWSM) *Audit study of Apple News as a sociotechnical news curation system (trending stories section).*
- [Auditing Algorithms: On Lessons Learned and the Risks of Data Minimization](https://dl.acm.org/doi/pdf/10.1145/3375627.3375852) - (AIES) *A practical audit for a well-being recommendation app developed by Telefónica (mostly on bias).*
- [Extracting Training Data from Large Language Models](https://arxiv.org/pdf/2012.07805) - (arxiv) *Performs a training data extraction attack to recover individual training examples by querying the language model.*

#### 2019
- [Adversarial Frontier Stitching for Remote Neural Network Watermarking](https://arxiv.org/abs/1711.01894) - (Neural Computing and Applications) [(Alternative implementation)](https://github.com/dunky11/adversarial-frontier-stitching) *Check if a remote machine learning model is a "leaked" one: through standard API requests to a remote model, extract (or not) a zero-bit watermark, that was inserted to watermark valuable models (eg, large deep neural networks).*
- [Knockoff Nets: Stealing Functionality of Black-Box Models](https://arxiv.org/abs/1812.02766.pdf) - (CVPR) *Ask to what extent can an adversary steal functionality of such "victim" models based solely on blackbox interactions: image in, predictions out.*
- [Opening Up the Black Box:Auditing Google's Top Stories Algorithm](https://par.nsf.gov/servlets/purl/10101277) - (Flairs-32) *Audit of the Google's Top stories panel that pro-vides insights into its algorithmic choices for selectingand ranking news publisher*
- [Making targeted black-box evasion attacks effective andefficient](https://arxiv.org/pdf/1906.03397.pdf) - (arXiv) *Investigates how an adversary can optimally use its query budget for targeted evasion attacks against deep neural networks.*
- [Online Learning for Measuring Incentive Compatibility in Ad Auctions](https://research.fb.com/wp-content/uploads/2019/05/Online-Learning-for-Measuring-Incentive-Compatibility-in-Ad-Auctions.pdf) - (WWW) *Measures the incentive compatible- (IC) mechanisms (regret) of black-box auction platforms.* 
- [TamperNN: Efficient Tampering Detection of Deployed Neural Nets](https://arxiv.org/abs/1903.00317) - (ISSRE) *Algorithms to craft inputs that can detect the tampering with a remotely executed classifier model.*
- [Neural Network Model Extraction Attacks in Edge Devicesby Hearing Architectural Hints](https://arxiv.org/pdf/1903.03916.pdf) - (arxiv) *Through the acquisition of memory access events from bus snooping, layer sequence identification bythe LSTM-CTC model, layer topology connection according to the memory access pattern, and layer dimension estimation under data volume constraints, it demonstrates one can accurately recover the a similar network architecture as the attack starting point*
- [Stealing Knowledge from Protected Deep Neural Networks Using Composite Unlabeled Data](https://ieeexplore.ieee.org/abstract/document/8851798) - (ICNN) *Composite method which can be used to attack and extract the knowledge ofa black box model even if it completely conceals its softmaxoutput.*
- [Neural Network Inversion in Adversarial Setting via Background Knowledge Alignment](https://dl.acm.org/citation.cfm?id=3354261) - (CCS) *Model inversion approach in the adversary setting based on training an inversion model that acts as aninverse of the original model. With no fullknowledge about the original training data, an accurate inversion is still possible by training the inversion model on auxiliary samplesdrawn from a more generic data distribution.*

#### 2018
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://arxiv.org/abs/1711.00399) - (Harvard Journal of Law & Technology) *To explain a decision on x, find a conterfactual: the closest point to x that changes the decision.*
- [Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation](https://arxiv.org/abs/1710.06169) - (AIES) *Treats black box models as teachers, training transparent student models to mimic the risk scores assigned by black-box models.*
- [Towards Reverse-Engineering Black-Box Neural Networks](https://arxiv.org/abs/1711.01768) - (ICLR) [(Code)](https://github.com/coallaoh/WhitenBlackBox) *Infer inner hyperparameters (eg number of layers, non-linear activation type) of a remote neural network model by analysing its response patterns to certain inputs.*
- [Data driven exploratory attacks on black box classifiers in adversarial domains](https://www.sciencedirect.com/science/article/pii/S092523121830136X) - (Neurocomputing) *Reverse engineers remote classifier models (e.g., for evading a CAPTCHA test).*
- [xGEMs: Generating Examplars to Explain Black-Box Models](https://arxiv.org/pdf/1806.08867.pdf) - (arXiv) *Searches bias in the black box model by training an unsupervised implicit generative model. Thensummarizes the black-box model behavior quantitatively by perturbing data samples along the data manifold.*
- [Learning Networks from Random Walk-Based Node Similarities](https://arxiv.org/pdf/1801.07386) - (NIPS) *Reversing graphs by observing some random walk commute times.*
- [Identifying the Machine Learning Family from Black-Box Models](https://rd.springer.com/chapter/10.1007/978-3-030-00374-6_6) - (CAEPIA) *Determines which kind of machine learning model is behind the returned predictions.* 
- [Stealing Neural Networks via Timing Side Channels](https://arxiv.org/pdf/1812.11720.pdf) - (arXiv) *Stealing/approximating a model through timing attacks using queries.*
- [Copycat CNN: Stealing Knowledge by Persuading Confession with Random Non-Labeled Data](https://arxiv.org/abs/1806.05476) - (IJCNN)  [(Code)](https://github.com/jeiks/Stealing_DL_Models) *Stealing black-box models (CNNs) knowledge by querying them with random natural images (ImageNet and Microsoft-COCO).*
- [Auditing the Personalization and Composition of Politically-Related Search Engine Results Pages](https://dl.acm.org/doi/10.1145/3178876.3186143) - (WWW) *A Chrome extension to survey participants and collect the Search Engine Results Pages (SERPs) and autocomplete suggestions, for studying personalization and composition.*

#### 2017
- [Uncovering Influence Cookbooks : Reverse Engineering the Topological Impact in Peer Ranking Services](https://dl.acm.org/authorize.cfm?key=N21772) - (CSCW) *Aims at identifying which centrality metrics are in use in a peer ranking service.*
- [The topological face of recommendation: models and application to bias detection](https://arxiv.org/abs/1704.08991) - (Complex Networks) *Proposes a bias detection framework for items recommended to users.*
- [Membership Inference Attacks Against Machine Learning Models](http://ieeexplore.ieee.org/document/7958568/) - (Symposium on Security and Privacy) *Given a machine learning model and a record, determine whether this record was used as part of the model's training dataset or not.*
- [Practical Black-Box Attacks against Machine Learning](https://dl.acm.org/citation.cfm?id=3053009) - (Asia CCS) *Understand how vulnerable is a remote service to adversarial classification attacks.*

#### 2016
- [Algorithmic Transparency via Quantitative Input Influence: Theory and Experiments with Learning Systems](https://www.andrew.cmu.edu/user/danupam/datta-sen-zick-oakland16.pdf) - (IEEE S&P) *Evaluate the individual, joint and marginal influence of features on a model using shapley values.*
- [Auditing Black-Box Models for Indirect Influence](https://arxiv.org/abs/1602.07043) - (ICDM) *Evaluate the influence of a variable on a black-box model by "cleverly" removing it from the dataset and looking at the accuracy gap*
- [Iterative Orthogonal Feature Projection for Diagnosing Bias in Black-Box Models](https://arxiv.org/abs/1611.04967) - (FATML Workshop) *Performs feature ranking to analyse black-box models*
- [Bias in Online Freelance Marketplaces: Evidence from TaskRabbit](http://datworkshop.org/papers/dat16-final22.pdf) - (dat workshop) *Measures the TaskRabbit's search algorithm rank.* 
- [Stealing Machine Learning Models via Prediction APIs](https://www.usenix.org/conference/usenixsecurity16/technical-sessions/presentation/tramer) - (Usenix Security)  [(Code)](https://github.com/ftramer/Steal-ML) *Aims at extracting machine learning models in use by remote services.*
- [“Why Should I Trust You?”Explaining the Predictions of Any Classifier](https://arxiv.org/pdf/1602.04938v3.pdf) - (arXiv)  [(Code)](https://github.com/marcotcr/lime-experiments) *Explains a blackbox classifier model by sampling around data instances.*
- [Back in Black: Towards Formal, Black Box Analysis of Sanitizers and Filters](http://ieeexplore.ieee.org/document/7546497/) - (Security and Privacy) *Black-box analysis of sanitizers and filters.*
- [Algorithmic Transparency via Quantitative Input Influence: Theory and Experiments with Learning Systems](http://ieeexplore.ieee.org/document/7546525/) - (Security and Privacy) *Introduces measures that capture the degree of influence of inputs on outputs of the observed system.*
- [An Empirical Analysis of Algorithmic Pricing on Amazon Marketplace](https://mislove.org/publications/Amazon-WWW.pdf) - (WWW) [(Code)](http://personalization.ccs.neu.edu) *Develops a methodology for detecting algorithmic pricing, and use it empirically to analyze their prevalence and behavior on Amazon Marketplace.*

#### 2015
- [Certifying and Removing Disparate Impact](https://arxiv.org/abs/1412.3756) - (SIGKDD) *Proposes SVM-based methods to certify absence of bias and methods to remove biases from a dataset.*
- [Peeking Beneath the Hood of Uber](https://dl.acm.org/citation.cfm?id=2815681) - (IMC) *Infer implementation details of Uber's surge price algorithm.*

#### 2014
- [A peek into the black box: exploring classifiers by randomization](https://link.springer.com/article/10.1007/s10618-014-0368-8) - (Data Mining and Knowledge Discovery journal) ([code](https://github.com/tsabsch/goldeneye)) *Finds groups of features that can be permuted without changing the output label of predicted samples*
- [XRay: Enhancing the Web's Transparency with Differential Correlation](https://www.usenix.org/node/184394) - (USENIX Security) *Audits which user profile data were used for targeting a particular ad, recommendation, or price.*

#### 2013
- [Measuring Personalization of Web Search](https://dl.acm.org/citation.cfm?id=2488435) - (WWW) *Develops a methodology for measuring personalization in Web search result.*
- [Auditing: Active Learning with Outcome-Dependent Query Costs](https://www.cs.bgu.ac.il/~sabatos/papers/SabatoSarwate13.pdf) - (NIPS) *Learns from a binary classifier paying only for negative labels.*

#### 2012
- [Query Strategies for Evading Convex-Inducing Classifiers](http://www.jmlr.org/papers/v13/nelson12a.html) - (JMLR) *Evasion methods for convex classifiers. Considers evasion complexity.*

#### 2008
- [Privacy Oracle: a System for Finding Application Leakswith Black Box Differential Testing](https://dl.acm.org/citation.cfm?id=1455806) - (CCS) *Privacy Oracle: a system that uncovers applications' leaks of personal information in transmissions to remote servers.*

#### 2005
- [Adversarial Learning](https://dl.acm.org/citation.cfm?id=1081950) - (KDD) *Reverse engineering of remote linear classifiers, using membership queries.*

### Related Events

#### 2025
* [AAAI workshop on AI Governance: Alignment, Morality, and Law](https://aaai.org/conference/aaai/aaai-25/workshop-list/#ws06)

#### 2024
* [1st International Conference on Auditing and Artificial Intelligence](https://www.ircg.msm.uni-due.de/ai/)
* [Regulatable ML Workshop (RegML'24)](https://regulatableml.github.io/)

#### 2023
* [Supporting User Engagement in Testing, Auditing, and Contesting AI (CSCW User AI Auditing)](https://cscw-user-ai-auditing.github.io/)
* [Workshop on Algorithmic Audits of Algorithms (WAAA)](https://algorithmic-audits.github.io)
* [Regulatable ML Workshop (RegML'23)](https://regulatableml.github.io/)

Algorithms

Algorithms 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Algorithms

Algorithms 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README