分批次训练和评估神经网络模型

【背景】 

训练神经网络模型的时候，特征组合太多，电脑的资源会不足，所以采用分批逐步进行。已经处理过的批次保存下来，在下一次跳过，只做新加入的批次训练。

选择最优模型组合在中间结果的范围内选择，这样能保证所有的特征都能得到组合，所有的组合都能得到训练和评估。

【流程】

+-------------------------------------+
|          开始 (Start)               |
+-------------------------------------+|v
+-------------------------------------+
| 读取中间结果 (loss_records)          |
+-------------------------------------+|v
+-------------------------------------+
| 计算总的特征组合数量               |
| (total_combinations)               |
+-------------------------------------+|v
+-------------------------------------+
| 计算批次数量 (num_batches)          |
+-------------------------------------+|v
+-------------------------------------+
| 初始化进度条                       |
+-------------------------------------+|v
+-------------------------------------+
| 清理多余记录                        |
| (Clean extra records)               |
+-------------------------------------+|v
+-------------------------------------+
| 遍历每个批次 (for each batch)       |
+-------------------------------------+|v
+-------------------------------------+
| 获取当前批次特征组合和数据          |
+-------------------------------------+|v
+-------------------------------------+
| 检查当前批次是否已处理              |
| (if batch in loss_records)          |
+-------------------+-----------------+
|       否          |        是       |
|                   |                 |
v                   |                 |
+-------------------------------------+|
| 调用 train_and_evaluate_torch        |
+-------------------------------------+||                      |v                      |
+-------------------------------------+|
| 更新所有评估结果                    | |
+-------------------------------------+ ||                     | vv                     +-------------------------------------+
+-------------------------------------+| 跳过已处理的批次，更新评估结果    |
| 保存中间结果                        |+-------------------------------------+
| (save intermediate results)         |
+-------------------------------------+|v
+-------------------------------------+
| 更新进度条                          |
+-------------------------------------+|v
+-------------------------------------+
| 所有批次处理完成                    |
| (All batches processed)             |
+-------------------------------------+|v
+-------------------------------------+
| 保存最佳模型和特征组合到Excel        |
| (save_result_to_excel)              |
+-------------------------------------+|v
+-------------------------------------+
|               结束 (End)            |
+-------------------------------------+

【需求】

读取中间结果
执行特征工程
遍历传入的特征组合

 对比中间结果和新传入的特征组合，
 找出和新传入的特征组合的差异，包括新增的和不再用的
 执行训练和评估，针对新增的，同步中间数据，中间结果中也包括预测值和模型参数（因为我希望从中选出最优模型，并记录，其中也包括参数信息和预测值）
 从最新的评估数据（包括新的和中间结果中的）， 选出最优的特征组合，保存到excel 

 【代码】

import os
import json
import pandas as pd
from tqdm import tqdm
import logging# 读取中间结果以防程序中途停止
loss_records = {}
if os.path.exists(loss_records_file):try:with open(loss_records_file, "r") as f:loss_records = json.load(f)print('~~~~~~~~从中间文件中读取到的loss_records：', loss_records)# 确保键是字符串，并转换回元组形式loss_records = {deserialize_features(k): v for k, v in loss_records.items()}print('~~~~~~~~转换回元组形式的loss_records：', loss_records)print("成功加载 loss_records.json")except json.JSONDecodeError as e:print(f"JSONDecodeError: {e}. 重置 loss_records.json 文件内容。")loss_records = {}with open(loss_records_file, "w") as f:json.dump(loss_records, f)# 获取所有特征组合的总数
total_combinations = len(feature_combinations)# 计算批次数量
num_batches = (total_combinations + combination_batch_size - 1) // combination_batch_size# 进度条初始化
pbar = tqdm(total=total_combinations, desc='特征组合训练进度', position=0, leave=True)
all_evaluation_results = []
new_feature_set = set(feature_combinations)# 删除 loss_records 中多余的记录
loss_records = {k: v for k, v in loss_records.items() if deserialize_features(k) in new_feature_set}
print('Cleaned loss_records:', loss_records)for batch_index in range(num_batches):start = batch_index * combination_batch_sizeend = min(start + combination_batch_size, total_combinations)current_batch = feature_combinations[start:end]current_normalized_data = normalized_data[start:end]print('current_batch: ', current_batch)print('loss_records: ', loss_records)# 检查当前批次是否已处理过if all(features in loss_records for features in current_batch):# 更新进度条pbar.update(len(current_batch))print('跳过已经处理过的批次')# 将已处理过的结果添加到所有评估结果中for features in current_batch:serialized_features = serialize_features(features)if serialized_features in loss_records:results = loss_records[serialized_features]all_evaluation_results.append({'features': features,'mse': results['MSE'],'mae': results['MAE'],'r2': results['R2']})continueprint('----没有跳过----已经处理过的批次')# 调用 train_and_evaluate_torch 函数处理当前批次的特征组合evaluation_results = train_and_evaluate_torch(current_batch, current_normalized_data, param_model, scaler_close, evaluation_results, n, data_obj, parameter_period, loss_records)all_evaluation_results.extend(evaluation_results)# 保存中间结果for features in current_batch:serialized_features = serialize_features(features)print(f'Serializing features: {features} -> {serialized_features}')# 提取结果并保存results = next(item for item in evaluation_results if item['features'] == features)if 'best_metrics' in results:best_metrics = results['best_metrics']loss_records[serialized_features] = {'MSE': convert_numpy_types(best_metrics['mse']),'MAE': convert_numpy_types(best_metrics['mae']),'R2': convert_numpy_types(best_metrics['r2'])}else:loss_records[serialized_features] = {'MSE': convert_numpy_types(results['mse']),'MAE': convert_numpy_types(results['mae']),'R2': convert_numpy_types(results['r2'])}# 输出当前的 loss_records 以进行调试print('Current loss_records before saving: ', loss_records)with open(loss_records_file, "w") as f:json.dump(loss_records, f)# 再次读取并检查文件内容，确保保存正确with open(loss_records_file, "r") as f:loaded_loss_records = json.load(f)print('Loaded loss_records after saving: ', loaded_loss_records)# 更新进度条pbar.update(len(current_batch))print("所有批次处理完成。")
pbar.close()# 最佳模型和每个特征组合的最佳模型保存到excel
save_result_to_excel(strategy_name, all_evaluation_results, OUTPUT_FILE_NEURAL_NETWORK_PATH, weights)def save_result_to_excel(strategy_name, evaluation_results, file_path, weights=None):"""数据保存到excel.Parameters:- evaluation_results 评估数据- file_path excel文件名称，用来保存测试报告Returns:None"""# print('评估数据evaluation_results:', evaluation_results)strategy_func = strategy_mapping.get(strategy_name)if strategy_func:num_params = len(inspect.signature(strategy_func).parameters)if weights and num_params > 1:best_result = strategy_func(evaluation_results, weights)print("best_result assigned successfully:", best_result)else:best_result = strategy_func(evaluation_results)print("best_result assigned successfully:", best_result)print('>>>>>>>>>>保存best_result>>>>>>>>>', best_result)print()    try:  # 创建一个空列表来存储评估过程的结果evaluation_process_data = []# 添加评估过程中的结果for result in evaluation_results:evaluation_process_data.append({'Features': result['features'],'Best Parameters': result['best_params'],'Best Metrics': result['best_metrics']})# 创建DataFrame来存储评估过程的结果df_evaluation_process = pd.DataFrame(evaluation_process_data)print('训练过程的数据：df_evaluation_process', df_evaluation_process)# 创建一个空的DataFrame来存储最佳模型的结果df_best_model_results = pd.DataFrame(columns=['Features', 'Best Predictions'])if best_result is not None:df_best_model_results.loc[0] = {'Features': best_result['features'],  # 使用best_result中的特征信息'Best Predictions': best_result['predictions']}# 倒置最佳模型结果DataFrame的行列df_best_model_results_transposed = df_best_model_results.transpose()# 创建一个新的 DataFrame，用于存储转置后的数据以及其含义df_with_labels = pd.DataFrame(columns=['Label', 'Value'])# 将原始表头作为索引，添加到新 DataFrame 中for feature in df_best_model_results_transposed.index:# 获取转置后数据的值，而不包括索引和数据类型信息value = df_best_model_results_transposed.loc[feature].values[0]df_with_labels = pd.concat([df_with_labels, pd.DataFrame({'Label': [feature], 'Value': [value]})], ignore_index=True)# 保存最佳模型的结果到Excel文件with pd.ExcelWriter(file_path, engine='xlsxwriter') as writer:df_with_labels.to_excel(writer, sheet_name='Best Model Results', index=False)print('执行了保存数据到excel,路径是:') print(file_path)    else:print("best_result is None, cannot save to excel")logging.error("best_result is None, cannot save to excel")except Exception as e:print(f"保存测试结果到excel: {e}")logging.error(f"save result to excel: {e}") else:print('Invalid strategy name:', strategy_name)

要点

1. 清理多余记录：在处理批次之前，根据新的特征组合清理 loss_records 中多余的记录。
2. 更新所有评估结果：即使跳过已处理的批次，也将其评估结果添加到 all_evaluation_results 中，以确保最终的最佳模型选择是基于所有特征组合。
3. 保存最佳结果到Excel：保持 save_result_to_excel 函数逻辑不变，确保从所有评估结果中选出最优模型并保存。

这样可以确保即使跳过了一些已处理的批次，最终的最优模型仍然是从所有特征组合中选出的，并且中间结果不会包含多余的记录。