Real-World Case Studies
We took real code from open-source projects and generated specs from it. Here's how it panned out: specs are 5-10x shorter than the code they describe, and equivalent code that passes all tests will be generated back from them.
| Case Study | Code LOC[1] | Spec LOC[1] | Shrink Factor | Tests Passed |
|---|---|---|---|---|
EML to .md converter for markitdown (Python library for converting anything to markdown) | 139 | 14 | 9.9x | before: 165 after: 192 (27 tests added) |
WebVTT subtitles support for yt-dlp (video downloader) | 255 | 38 | 6.7x | before: 1241/1242 after: 1278/1279 (37 tests added) |
Italian SSN generator for Faker (python library for generating mock data) | 165[2] | 21 | 7.9x | before: 2216 after: 2229 (13 tests added) |
Encoding auto-detection and normalization for beautifulsoup4 (Python library for parsing HTML and XML) | 826 | 141 | 5.9x | before: 889 after: 914 (25 tests added) |
[1] When computing LOC, we strip blank lines and break long lines into many
[2] List of Italian municipalities codes (~8000 LOC) is excluded
[2] List of Italian municipalities codes (~8000 LOC) is excluded