HuggingFace datasets - 下载数据

---

下载数据

1、Python 代码下载

from datasets import load_dataset
imdb = load_dataset("imdb") 
# name参数为full或mini，full表示下载全部数据，mini表示下载部分少量数据
# dataset = load_dataset(model_name, name="full") 

---

imdb
'''
DatasetDict({train: Dataset({features: ['text', 'label'],num_rows: 25000})test: Dataset({features: ['text', 'label'],num_rows: 25000})unsupervised: Dataset({features: ['text', 'label'],num_rows: 50000})
})
'''

默认保存在 ~/.cache/huggingface 文件夹

数据格式如下：

$ cd datasets/imdb/
$ tree
.
└── plain_text└── 0.0.0├── e6281661ce1c48d982bc483cf8a173c1bbeb5d31│   ├── dataset_info.json│   ├── imdb-test.arrow│   ├── imdb-train.arrow│   └── imdb-unsupervised.arrow├── e6281661ce1c48d982bc483cf8a173c1bbeb5d31.incomplete_info.lock└── e6281661ce1c48d982bc483cf8a173c1bbeb5d31_builder.lock3 directories, 6 files

---

2、huggingface-cli 命令下载
这样下载也会保存到 ~/.cache/huggingface 文件夹

huggingface-cli download --repo-type dataset imdb

---

3、git

---

修改默认保存地址 TRANSFORMERS_CACHE

环境变量添加

export TRANSFORMERS_CACHE='path/'

---

代码中使用

import os 
os.environ['TRANSFORMERS_CACHE']=''

---

保存到本地 & 本地加载

保存

save_path = '/Users/xx/Downloads/imdb' 
imdb.save_to_disk(save_path)
'''
Saving the dataset (1/1 shards): 100%|█| 25000/25000 [00:00<00:00, 97903.42 exam
Saving the dataset (1/1 shards): 100%|█| 25000/25000 [00:00<00:00, 251032.07 exa
Saving the dataset (1/1 shards): 100%|█| 50000/50000 [00:00<00:00, 88591.53 exam
'''imdb2 = load_from_disk(save_path)
imdb2
'''
DatasetDict({train: Dataset({features: ['text', 'label'],num_rows: 25000})test: Dataset({features: ['text', 'label'],num_rows: 25000})unsupervised: Dataset({features: ['text', 'label'],num_rows: 50000})
})
'''

---

存储格式如下：

$ cd imdb/
$ tree
.
├── dataset_dict.json
├── test
│   ├── data-00000-of-00001.arrow
│   ├── dataset_info.json
│   └── state.json
├── train
│   ├── data-00000-of-00001.arrow
│   ├── dataset_info.json
│   └── state.json
└── unsupervised├── data-00000-of-00001.arrow├── dataset_info.json└── state.json3 directories, 10 files

---

加载

# 指定加载测试集
save_path1 = '/Users/xx/Downloads/imdb/test' 
imdb3 = load_from_disk(save_path1)
imdb3
'''
Dataset({features: ['text', 'label'],num_rows: 25000
})
'''imdb4 = load_dataset('imdb') # 默认加载 `.cache` 中的数据 imdb4 = load_dataset(path='/Users/xx/Downloads/imdb')
'''
Generating train split: 1 examples [00:00, 69.32 examples/s]
Generating test split: 1 examples [00:00, 277.31 examples/s]
'''
imdb4
'''
DatasetDict({train: Dataset({features: ['_data_files', '_fingerprint', '_format_columns', '_format_kwargs', '_format_type', '_output_all_columns', '_split'],num_rows: 1})test: Dataset({features: ['_data_files', '_fingerprint', '_format_columns', '_format_kwargs', '_format_type', '_output_all_columns', '_split'],num_rows: 1})
})
'''# 指定加载文件 - 失败
save_path2 = '/Users/xx/Downloads/imdb/test/data-00000-of-00001.arrow' 
imdb4 =  load_from_disk(save_path2)
'''
Traceback (most recent call last):File "<stdin>", line 1, in <module>File "/Users/xx/miniconda3/lib/python3.11/site-packages/datasets/load.py", line 2215, in load_from_diskraise FileNotFoundError(
FileNotFoundError: Directory /Users/xx/Downloads/imdb/test/data-00000-of-00001.arrow is neither a `Dataset` directory nor a `DatasetDict` directory.
'''

---

无法从 .cache/huggingface/datasets 加载

path = '/Users/xx/.cache/huggingface/datasets/imdb' 
from datasets import load_from_diskimdb2 = load_from_disk(path)
'''
Traceback (most recent call last):File "<stdin>", line 1, in <module>File "/Users/xx/miniconda3/lib/python3.11/site-packages/datasets/load.py", line 2215, in load_from_diskraise FileNotFoundError(
FileNotFoundError: Directory /Users/xx/.cache/huggingface/datasets/imdb is neither a `Dataset` directory nor a `DatasetDict` directory.
'''path1 = '/Users/xx/.cache/huggingface/datasets/imdb/plain_text/0.0.0/e6281661ce1c48d982bc483cf8a173c1bbeb5d31/imdb-test.arrow'  imdb2 = load_from_disk(path1)
'''
Traceback (most recent call last):File "<stdin>", line 1, in <module>File "/Users/xx/miniconda3/lib/python3.11/site-packages/datasets/load.py", line 2215, in load_from_diskraise FileNotFoundError(
FileNotFoundError: Directory /Users/xx/.cache/huggingface/datasets/imdb/plain_text/0.0.0/e6281661ce1c48d982bc483cf8a173c1bbeb5d31/imdb-test.arrow is neither a `Dataset` directory nor a `DatasetDict` directory.
'''path1 = '/Users/xx/.cache/huggingface/datasets/imdb/plain_text/0.0.0/e6281661ce1c48d982bc483cf8a173c1bbeb5d31/' 
imdb2 = load_from_disk(path1)
'''
Traceback (most recent call last):File "<stdin>", line 1, in <module>File "/Users/xx/miniconda3/lib/python3.11/site-packages/datasets/load.py", line 2215, in load_from_diskraise FileNotFoundError(
FileNotFoundError: Directory /Users/xx/.cache/huggingface/datasets/imdb/plain_text/0.0.0/e6281661ce1c48d982bc483cf8a173c1bbeb5d31/ is neither a `Dataset` directory nor a `DatasetDict` directory.
'''path1 = '/Users/xx/.cache/huggingface/datasets/imdb/plain_text/0.0.0/' imdb2 = load_from_disk(path1)
Traceback (most recent call last):File "<stdin>", line 1, in <module>File "/Users/xx/miniconda3/lib/python3.11/site-packages/datasets/load.py", line 2215, in load_from_diskraise FileNotFoundError(
FileNotFoundError: Directory /Users/xx/.cache/huggingface/datasets/imdb/plain_text/0.0.0/ is neither a `Dataset` directory nor a `DatasetDict` directory.path1 = '/Users/xx/.cache/huggingface/datasets/imdb/plain_text/' 
imdb2 = load_from_disk(path1)
'''
Traceback (most recent call last):File "<stdin>", line 1, in <module>File "/Users/xx/miniconda3/lib/python3.11/site-packages/datasets/load.py", line 2215, in load_from_diskraise FileNotFoundError(
FileNotFoundError: Directory /Users/xx/.cache/huggingface/datasets/imdb/plain_text/ is neither a `Dataset` directory nor a `DatasetDict` directory.
'''

---

读取 .arrow 数据

双击 .arrow 文件无法直接查看，使用下面代码可以查看内容

def read_arrow_to_df_julia_ok(path):with open(path, "rb") as f:r = pyarrow.ipc.RecordBatchStreamReader(f)df = r.read_pandas()return dfpath = '/Users/xx/Downloads/imdb/test/data-00000-of-00001.arrow'
path = '/Users/xx/.cache/huggingface/datasets/imdb/plain_text/0.0.0/e6281661ce1c48d982bc483cf8a173c1bbeb5d31/imdb-test.arrow'
table = read_arrow_to_df_julia_ok(path)
# 打印数据
print('打印数据：\n', table)

结果

打印数据：text  label
0      I love sci-fi and am willing to put up with a ...      0
1      Worth the entertainment value of a rental, esp...      0
2      its a totally average film with a few semi-alr...      0
3      STAR RATING: ***** Saturday Night **** Friday ...      0
4      First off let me say, If you haven't enjoyed a...      0
...                                                  ...    ...
24995  Just got around to seeing Monster Man yesterda...      1
24996  I got this as part of a competition prize. I w...      1
24997  I got Monster Man in a box set of three films ...      1
24998  Five minutes in, i started to feel how naff th...      1
24999  I caught this movie on the Sci-Fi channel rece...      1

---