img2dataset
Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.
Language: python
Author: GitHub Repos (@github-repos)
0 stars · 5 views
Viewing path: mypy.ini
Files
- resize_no.jpg (image)
- blur_test_files (txt)
- doc_assets (txt)
- notebook (txt)
- llm_doc (txt)
- dataset_examples (txt)
- .github (github)
- workflows (txt)
- resize_test_image (txt)
- test_files (txt)
- laion-face.md (md)
- test_writer.py (py)
- datacomp.md (md)
- requirements-test.txt (txt)
- CLAUDE.md (md)
- Makefile (txt)
- common_pool.md (md)
- SBUcaptions.md (md)
- cc12m.md (md)
- laion-aesthetic.md (md)
- cc3m.md (md)
- laion-high-resolution.md (md)
- coyo-700m.md (md)
- mscoco.md (md)
- laion-coco.md (md)
- README.md (md)
- .gitpod.yml (yml)
- mypy.ini (ini)
- LICENSE (txt)
- examples (txt)
- distributed_img2dataset_tutorial.md (md)
- ray_example (txt)
- simple_example.py (py)
- pyspark_example.py (py)
- cluster_minimal.yaml (yaml)
- img2dataset (txt)
- ray_example.py (py)
- reader.py (py)
- main.py (py)
- __init__.py (py)
- writer.py (py)
- blurrer.py (py)
- distributor.py (py)
- logger.py (py)
- downloader.py (py)
- architecture.md (md)
- ci.yml (yml)
- tests (txt)
- dependabot.yml (yml)
- test_resizer.py (py)
- regenerate_blur_references.py (py)
- 416_264.jpg (image)
- 389_535.jpg (image)
- 123_456.jpg (image)
- large_bench_tf.sh (sh)
- large_bench.sh (sh)
- sample_image.txt (txt)
- test_downloader.py (py)
- fixtures.py (py)
- test_reader.py (py)
- resize_keep_ratio.jpg (image)
- original.png (image)
- resize_center_crop.jpg (image)
- resize_keep_ratio_largest.jpg (image)
- HISTORY.md (md)
- http_server.py (py)
- wandb_table.png (image)
- wandb_metrics.png (image)
- ATTRIBUTION.md (markdown)
- 321_421.jpg (image)
- python-publish.yml (yml)
- 208_495.jpg (image)
- 456_123.jpg (image)
- test_blurrer.py (py)
- setup.py (py)
- blurred.png (image)
- unique_images.txt (txt)
- README.md (md)
- resizer.py (py)
- laion-art.md (md)
- conftest.py (py)
- resize_border.jpg (image)
- test_1000.txt (txt)
- benchmark.sh (sh)
- .gitpod.DockerFile (dockerfile)
- hashes.json (json)
- 524_316.jpg (image)
- laion5B.md (md)
- requirements.txt (txt)
- test_main.py (py)
- s3_bench.sh (sh)
- laion400m.md (md)
