Tensorflow2如何读取自制数据集并训练模型？-- Tensorflow自学笔记13

一. 如何自制数据集？

1. 目录结构

以下是自制数据集-手写数字集, 保存在目录 mnist_image_label 下

2. 数据存储格式 

2.1. 目录mnist_train_jpeg_60000 下存放的是 60000张用于测试的手写数字

       如 : 0_5.jpg, 表示编号为0，标签为5的图片

              6_1.jpg, 表示编号为6，标签为1的图片

2.2. 目录mnist_test_jpeg_10000 下存放的是10000张用于测试的手写数字

        图片存储格式与1.1相同

2.3. txt文件 mnist_train_jpg_60000.txt，里面存放的是

     

        比如，第一行  28755_0.jpg   0     前面表示图片名称，后面的0表示该图片对应的标签，这里表示该图片是手写数字0.

2.4. txt文件 mnist_test_jpg_10000.txt   , 存放的是测试数据集的标签

二. 如何读取自制数据集并输入神经网络

以下是test.py 如何读取自制数据集代码

1. 导入需要的库

import tensorflow as tf

 from PIL import Image

import numpy as np

import os

2.设置数据集所在文件目录 

   (test.py, 需和mnist_image_label 目录在同一级目录下)

train_path = './mnist_image_label/mnist_train_jpg_60000/'train_txt = './mnist_image_label/mnist_train_jpg_60000.txt'x_train_savepath = './mnist_image_label/mnist_x_train.npy'y_train_savepath = './mnist_image_label/mnist_y_train.npy'test_path = './mnist_image_label/mnist_test_jpg_10000/'test_txt = 'v/mnist_image_label/mnist_test_jpg_10000.txt'x_test_savepath = './mnist_image_label/mnist_x_test.npy' #训练集输入特征存储文件npy,y_test_savepath = './mnist_image_label/mnist_y_test.npy' #训练集标签存储文件

3.定义读取数据的函数

def generateds(path, txt):f = open(txt, 'r') # 以只读形式打开txt文件contents = f.readlines() # 读取文件中所有行f.close() # 关闭txt文件x, y_ = [], [] # 建立空列表for content in contents: # 逐行取出value = content.split() # 以空格分开，图片路径为value[0] , 标签为value[1] , 存入列表img_path = path + value[0] # 拼出图片路径和文件名print('image path....: '+img_path)img = Image.open(img_path) # 读入图片img = np.array(img.convert('L')) # 图片变为8位宽灰度值的np.array格式img = img / 255. # 数据归一化 （实现预处理）x.append(img) # 归一化后的数据，贴到列表xy_.append(value[1]) # 标签贴到列表y_print('loading : ' + content) # 打印状态提示x = np.array(x) # 变为np.array格式y_ = np.array(y_) # 变为np.array格式y_ = y_.astype(np.int64) # 变为64位整型return x, y_ # 返回输入特征x，返回标签y_

4.调用定义的函数

if os.path.exists(x_train_savepath) and os.path.exists(y_train_savepath) and os.path.exists(x_test_savepath) and os.path.exists(y_test_savepath):print('-------------Load Datasets-----------------')x_train_save = np.load(x_train_savepath)y_train = np.load(y_train_savepath)x_test_save = np.load(x_test_savepath)y_test = np.load(y_test_savepath)x_train = np.reshape(x_train_save, (len(x_train_save), 28, 28))x_test = np.reshape(x_test_save, (len(x_test_save), 28, 28))else:print('-------------Generate Datasets-----------------')x_train, y_train = generateds(train_path, train_txt)x_test, y_test = generateds(test_path, test_txt)print('-------------Save Datasets-----------------')x_train_save = np.reshape(x_train, (len(x_train), -1))x_test_save = np.reshape(x_test, (len(x_test), -1))np.save(x_train_savepath, x_train_save)np.save(y_train_savepath, y_train)np.save(x_test_savepath, x_test_save)np.save(y_test_savepath, y_test)

5. 搭建神经网络训练数据

model = tf.keras.models.Sequential([tf.keras.layers.Flatten(),tf.keras.layers.Dense(128, activation='relu'),tf.keras.layers.Dense(10, activation='softmax')
])model.compile(optimizer='adam',loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),metrics=['sparse_categorical_accuracy'])model.fit(x_train, y_train, batch_size=32, epochs=5, validation_data=(x_test, y_test), validation_freq=1)
model.summary()