ungoliant
:spider: The pipeline for the OSCAR corpus
Language: rust
Author: GitHub Repos (@github-repos)
0 stars · 0 views
Files
- img (txt)
- logo.png (image)
- README.md (md)
- benches (txt)
- processing (txt)
- rebuild.rs (rs)
- package.rs (rs)
- check.rs (rs)
- mod.rs (rs)
- identifiers (txt)
- multilingual.rs (rs)
- identification.rs (rs)
- model.rs (rs)
- mod.rs (rs)
- tag_convert.rs (rs)
- CODE_OF_CONDUCT.md (md)
- feature_request.md (md)
- bug_report.md (md)
- tests (txt)
- annotate_noisy.rs (rs)
- pipeline_bench.rs (rs)
- pipeline_bench_rayon.rs (rs)
- fasttext_bench.rs (rs)
- LICENSE (txt)
- .github (github)
- workflows (txt)
- rust.yml (yml)
- dependabot.yml (yml)
- ISSUE_TEMPLATE (txt)
- oscardoc_rebuild.rs (rs)
- res (txt)
- src (txt)
- download.rs (rs)
- lib.rs (rs)
- filtering (txt)
- filter.rs (rs)
- record.rs (rs)
- mod.rs (rs)
- sentence.rs (rs)
- transformers (txt)
- kenlm (txt)
- lru.rs (rs)
- adult_content.rs (rs)
- mod.rs (rs)
- sentence_filter.rs (rs)
- transform.rs (rs)
- mod.rs (rs)
- tiny.rs (rs)
- noisy.rs (rs)
- annotate.rs (rs)
- content_detector.rs (rs)
- category.rs (rs)
- header.rs (rs)
- commoncrawl (txt)
- shard.rs (rs)
- mod.rs (rs)
- mod.rs (rs)
- cli.rs (rs)
- lsh.rs (rs)
- io (txt)
- langfiles.rs (rs)
- mod.rs (rs)
- error.rs (rs)
- main.rs (rs)
- pipelines (txt)
- oscardoc (txt)
- pipeline.rs (rs)
- mod.rs (rs)
- types (txt)
- rebuild.rs (rs)
- mod.rs (rs)
- location.rs (rs)
- pipeline.rs (rs)
- mod.rs (rs)
- sources (txt)
- Cargo.toml (toml)
- ATTRIBUTION.md (markdown)
