fmujie - Torch
type: channel
link: https://blog.fmujie.cn/index.php/category/Torch/
description: PyTorch
items: 从A主机快速迁移Conda环境至B主机
items: 训练常用命令
items: 16、卷积残差模块算子融合
items: Linux 系统下torch的预训练模型保存路径
items: 15、Dropout原理以及Torch源码的实现
items: 14、保存与加载Pytorch训练的模型和超参
items: 9、PyTorch的nn.Sequential及ModuleList源码
items: 8、PyTorch的state_dict、parameters、modules源码
items: 24、手写并验证向量内积实现PyTorch二维卷积
items: 23、手写并验证滑动相乘实现PyTorch二维卷积

从A主机快速迁移Conda环境至B主机
type: item
date: 2025-11-25T09:04:00+08:00
description: A主机查看当前conda环境(base) ytt@ytt-System-Product-Name:~$ conda env list # conda environments: # base * /home/ytt/anaconda3 DCEvo /home/ytt/anaconda3/envs/DCEvo FAF /home/ytt/anaconda3/envs/FAF # 要迁移的环境 SegMiF /home/ytt/anaconda3/envs/SegMiF TDFusion /home/ytt/anaconda3/envs/TDFusion TIF /home/ytt/anaconda3/envs/TIF habitat /home/ytt/anaconda3/envs/habitat matcha /home/ytt/anaconda3/envs/matcha pytorch20 /home/ytt/anaconda3/envs/pytorch20 tardal /home/ytt/anaconda3/envs/tardal ylsnet /home/ytt/anaconda3/envs/ylsnet yolov5 /home/ytt/anaconda3/envs/yolov5 zjl /home/ytt/anaconda3/envs/zjls打包要迁移的环境(base) ytt@ytt-System-Product-Name:~$ cd /home/ytt/anaconda3/envs # 进入目录 (base) ytt@ytt-System-Product-Name:~$ tar -czvf FAF.tar.gz ./FAF/* # 打包=>使用FTP传到主机BB主机解压到当前目录(base) excelsior@dlu:~/miniconda3/envs$ tar -xzvf FAF.tar.gz # 解压到当前目录查看环境(base) excelsior@dlu:~/miniconda3/envs$ conda env list # conda environments: # base * /home/excelsior/miniconda3 FAF /home/excelsior/miniconda3/envs/FAF # 新解压出来的 INF /home/excelsior/miniconda3/envs/INF IR /home/excelsior/miniconda3/envs/IR MG /home/excelsior/miniconda3/envs/MG MM /home/excelsior/miniconda3/envs/MM MoLA /home/excelsior/miniconda3/envs/MoLA fmujie /home/excelsior/miniconda3/envs/fmujie mld /home/excelsior/miniconda3/envs/mld ylsnet /home/excelsior/miniconda3/envs/ylsnet激活环境并使用(base) excelsior@dlu:~/miniconda3/envs$ conda activate FAF无法使用pip/pip3安装包(FAF) excelsior@dlu:~/miniconda3/envs$ pip list -bash: /home/excelsior/miniconda3/envs/FAF/bin/pip：/home/ytt/anaconda3/envs/FAF/bin/python：解释器错误: 没有那个文件或目录 (FAF) excelsior@dlu:~/miniconda3/envs$ pip3 list -bash: /home/excelsior/miniconda3/envs/FAF/bin/pip3：/root/anaconda3/envs/FAF/bin/python：解释器错误: 权限不够解决方法编辑迁移环境bin目录下的pip文件，pip3同理文件内容#!/home/ytt/anaconda3/envs/FAF/bin/python # -*- coding: utf-8 -*- import re import sys from pip._internal.cli.main import main if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])修改第一行内容至新的环境如下#!/home/excelsior/minicond…

训练常用命令
type: item
date: 2024-08-30T23:44:00+08:00
description: 训练常用命令nohup python train.py > outLog.log &nohup挂后台运行，退出Xshell连接无影响，防止断网> outLog.log将输出输出到outLog.log文件&追加模式一般运行该命令后会提示如下：(base) root@d9dcc1730df7:~/data1/fmujie/dpl/cal_fgvc# nohup python train_distributed.py > outNA2245628.log & [1] 14131 (base) root@d9dcc1730df7:~/data1/fmujie/dpl/cal_fgvc# nohup: ignoring input and redirecting stderr to stdout如果忘记这个进程号也没关系ps -aux | grep train_distributed.py查看(base) root@d9dcc1730df7:~/data1/fmujie/dpl/cal_fgvc# ps -aux | grep train_distributed.py root 14133 104 0.5 34309892 1401112 pts/0 Rl 23:32 0:54 python train_distributed.py --dataset=nabirds --img_size=224 --lr_scale_ratio=4 root 14196 16.3 0.2 16130636 607052 pts/0 Rl 23:32 0:07 python train_distributed.py --dataset=nabirds --img_size=224 --lr_scale_ratio=4 root 14259 15.7 0.2 16130648 607052 pts/0 Sl 23:32 0:07 python train_distributed.py --dataset=nabirds --img_size=224 --lr_scale_ratio=4 root 14322 15.7 0.2 16130660 607116 pts/0 Sl 23:32 0:07 python train_distributed.py --dataset=nabirds --img_size=224 --lr_scale_ratio=4 root 14385 18.5 0.2 16130672 607072 pts/0 Sl 23:32 0:08 python train_distributed.py --dataset=nabirds --img_size=224 --lr_scale_ratio=4 root 14592 0.0 0.0 3296 716 pts/0 S+ 23:33 0:00 grep --color=auto train_distributed.py

16、卷积残差模块算子融合
type: item
date: 2024-07-27T20:43:00+08:00
description: R-Drop: Regularized Dropout for Neural NetworksDropout is a powerful and widely used technique to regularize the training of deep neural networks. Dropout在训练和推理时存在不一致的问题（集成学习）R 对每个子模型的分布做一个KL散度import numpy as np def train_r_drop(ratio, x, w1, b1, w2, b2): # 输入复制一份 x = torch.cat([x, x], dim=0) layer1 = np.maximum(0, np.dot(w1, x) + b1) mask1 = np.random.binomial(1, 1-ratio, layer1.shape) layer1 = layer1 * mask1 layer2 = np.maximum(0, np.dot(w2, layer1) + b2) mask2 = np.random.binomial(1, 1-ratio, layer2.shape) layer2 = layer2 * mask2 logits = func(layer2) logits1, logits2 = logits[:bs, :], logits[bs:, :] nll1 = nll(logits1, label) nll2 = nll(logits2, label) kl_los = kl(logits1, logits2) loss = nll1 + nll2 + kl_loss return loss def test(ratio, x, w1, b1, w2, b2): layer1 = np.maximum(0, np.dot(w1, x) + b1) layer1 = layer1 * (1-ratio) layer2 = np.maximum(0, np.dot(w2, layer1) + b2) layer2 = layer2 * (1-ratio) return layer2import torch conv_layer = torch.nn.Conv2d(in_channels=2, out_channels=2, kernel_size=3, padding="same") for i in conv_layer.named_parameters(): print(i) ''' ('weight', Parameter containing: tensor([[[[-0.1885, 0.1133, -0.0103], [-0.0462, -0.1434, -0.0569], [ 0.1969, 0.0827, -0.1245]], [[ 0.1554, 0.1459, 0.0405], [-0.0806, 0.1438, -0.1319], [-0.1495, -0.0960, 0.2193]]], [[[-0.1748, 0.1517, 0.0040], [ 0.0097, 0.1565, 0.1850], [-0.0261, 0.1298, 0.0211]], [[-0.0964, -0.0133, -0.0378], [ 0.0617, -0.2037, 0.0011], [-0.0542, -0.1499, 0.0187]]]], requires_grad=True)) ('bias', Parameter containing: tensor([0.0560, 0.2200], requires_grad=True)) ''' conv_layer.weights.size() ''' 对于输出的第1个通道而言： 对于输入的2个通道都有3×3的卷积核分别在这2个输入通道上做卷积再求和，就得到了，第1个输出通道的特征值。 对于输出的第2个通道而言： 又来了2个卷积核，对输入的2个通道做卷积，得到第2个输出通道的特征值。 torch.Size([2, 2, 3, 3]) 3,3为Kernel_size大小 2,2 是输出×输入 ''' conv_layer.bias.size() ''' torch.Size([2]) 和输出通道一致 ''' conv_layer = torch.nn.Conv2d(in_channels=2, out_channels=4, kernel_size=3, padding="same") print(conv_layer.weight.size()) ''' 4中每个输出通道，都有2套3×3的矩阵对输入进行滑动卷积，再将2个…

Linux 系统下torch的预训练模型保存路径
type: item
date: 2024-07-18T10:00:00+08:00
description: home\.cache\torch\checkpoints有什么用，遇到就知道了，有时候集群网不好，可以本地下载后传到集群上的指定目录，就不用在集群上龟速下载了。

15、Dropout原理以及Torch源码的实现
type: item
date: 2024-06-23T20:39:00+08:00
description: NN.DROPOUTCLASStorch.nn.Dropout(p=0.5, inplace=False)Parametersp (float) – probability of an element to be zeroed. Default: 0.5inplace (bool) – If set to True, will do this operation in-place. Default: FalseShape:Input: (∗)(∗). Input can be of any shapeOutput: (∗)(∗). Output is of the same shape as inputm = nn.Dropout(p=0.2) input = torch.randn(20, 16) output = m(input)如何判断当前是否为Train？ 参数中无training 参数 ==resolution：因为类的父类继承了module类，而module类种有self.training的定义，所以不需要再额外需要training参数了==TORCH.NN.FUNCTIONAL.DROPOUTtorch.nn.functional.dropout(input, p=0.5, training=True, inplace=False)Parametersp (float) – probability of an element to be zeroed. Default: 0.5training (bool) – apply dropout if is True. Default: Trueinplace (bool) – If set to True, will do this operation in-place. Default: FalseReturn typeTensor==函数无继承关系，所以需要额外的training参数，不过该函数还是底层，类内部也需要调用该参数，用的training参数为父父类中定义的==含dropout，训练时相当于集成学习，训练多个网络，而在测试时只是逼近需要在测试期间与训练阶段的期望值相互逼近的话，需要在测试阶段将权重$w$乘以一个$p$。因为在训练时按照$p$的概率保留了一些神经元的连接，所以在测试的时候需要在权重上乘以概率。但是为了在测试时耗时尽可能少，所以在训练时的权重上乘了一个放缩因子（实际运行中）。from .module import Module from .. import functional as F from torch import Tensor __all__ = ['Dropout', 'Dropout1d', 'Dropout2d', 'Dropout3d', 'AlphaDropout', 'FeatureAlphaDropout'] class _DropoutNd(Module): __constants__ = ['p', 'inplace'] p: float inplace: bool def __init__(self, p: float = 0.5, inplace: bool = False) -> None: super().__init__() if p < 0 or p > 1: raise ValueError(f"dropout probability has to be between 0 and 1, but got {p}") self.p = p self.inplace = inplace def extra_repr(self) -> str: return f'p={self.p}, inplace={self.inplace}' [docs]class Dropout(_DropoutNd): def forward(self, input: Tensor) -> Tensor: return F.dropout(input, self.p, self.training, self.inplace) [docs]class Dropout1d(_DropoutNd): # 仅有一个Tensor参数，无显式training参数 def forward(self, input: Tensor) -> Tensor: # 调用了函数 self.training哪来的？Class module中 return F.dropout1d(input, self.p, self.training, self.inplace) [docs]class Dropout2d(_DropoutNd): def forward…

14、保存与加载Pytorch训练的模型和超参
type: item
date: 2024-06-06T20:35:00+08:00
description: Saving and loading models for inference in PyTorchThere are two approaches for saving and loading models for inference in PyTorch.The first is saving and loading the state_dictand the second is saving and loading the entire model. def state_dict(self, *args, destination=None, prefix='', keep_vars=False): r"""Returns a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to ``None`` are not included. """ # TODO: Remove `args` and the parsing logic when BC allows. if len(args) > 0: if destination is None: destination = args[0] if len(args) > 1 and prefix == '': prefix = args[1] if len(args) > 2 and keep_vars is False: keep_vars = args[2] # DeprecationWarning is ignored by default warnings.warn( "Positional args are being deprecated, use kwargs instead. Refer to " "https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict" " for details.") if destination is None: destination = OrderedDict() destination._metadata = OrderedDict() local_metadata = dict(version=self._version) if hasattr(destination, "_metadata"): destination._metadata[prefix[:-1]] = local_metadata for hook in self._state_dict_pre_hooks.values(): hook(self, prefix, keep_vars) self._save_to_state_dict(destination, prefix, keep_vars) for name, module in self._modules.items(): if module is not None: module.state_dict(destination=destination, prefix=prefix + name + '.', keep_vars=keep_vars) for hook in self._state_dict_hooks.values(): hook_result = hook(self, destination, prefix, local_metadata) if hook_result is not None: destination = hook_result return destinationSteps构建DatasetDefine and initialize the neural networkclass Net(nn.Module): def __init__(self): super(Net, self).__init__() self.conv1 = nn.Conv2d(3, 6, 5) self.pool = nn.MaxPool2d(2, 2) self.conv2 = nn.Conv2d(6, 16, 5) self.fc1 = nn.Linear(16 * 5 * 5,…

9、PyTorch的nn.Sequential及ModuleList源码
type: item
date: 2024-03-31T20:20:00+08:00
description: train# 实例化一个模型,在模型后调用.train(True),说明我们将该模型设置为训练模式 def train(self: T, mode: bool = True) -> T: r"""Sets the module in training mode. This has any effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. :class:`Dropout`, :class:`BatchNorm`, etc. Args: mode (bool): whether to set training mode (``True``) or evaluation mode (``False``). Default: ``True``. Returns: Module: self """ # 首先判断mode是否为bool类型 if not isinstance(mode, bool): raise ValueError("training mode is expected to be boolean") self.training = mode # 如果传入了True,则self.training为True,且子模块递归都设置为训练模式 for module in self.children(): module.train(mode) return selfeval()def eval(self: T) -> T: r"""Sets the module in evaluation mode. This has any effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. :class:`Dropout`, :class:`BatchNorm`, etc. This is equivalent with :meth:`self.train(False) <torch.nn.Module.train>`. See :ref:`locally-disable-grad-doc` for a comparison between `.eval()` and several similar mechanisms that may be confused with it. Returns: Module: self """ return self.train(False)requires_grad_# 是否需要自动微分 module的函数 def requires_grad_(self: T, requires_grad: bool = True) -> T: r"""Change if autograd should record operations on parameters in this module. This method sets the parameters' :attr:`requires_grad` attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See :ref:`locally-disable-grad-doc` for a comparison between `.requires_grad_()` and several similar mechanisms that may be confused with it. Args: requires_grad (bool): whether autograd should record operations on parameters in this module. Default: ``True``. Returns: Module: self """ # 对当前模型的所有参数遍历 for p in self.parameters(): # 将每个参数的requires_g…

8、PyTorch的state_dict、parameters、modules源码
type: item
date: 2024-03-31T16:07:00+08:00
description: Save and Load the ModelPyTorch models store the learned parameters in an internal state dictionary, called state_dict. These can be persisted via the torch.save method:model = models.vgg16(weights='IMAGENET1K_V1') # 保存模型的参数、优化器状态、batch_nomalization、drop_out等等一系列的buffer变量 # model.state_dict() 只保留模型的权重 torch.save(model.state_dict(), 'model_weights.pth') # 每个model都会包含一个state_dict()状态字典:包含模型所有的参数和buffer变量To load model weights, you need to create an instance of the same model first, and then load the parameters using load_state_dict() method.model = models.vgg16() # we do not specify ``weights``, i.e. create untrained model model.load_state_dict(torch.load('model_weights.pth')) # 推理模式:会影响到drop_out、batch_nomalization model.eval()同时保存优化器权重的methodimport torch import torch.nn as nn import torch.optim as optim class Net(nn.Module): def __init__(self): super(Net, self).__init__() self.conv1 = nn.Conv2d(3, 6, 5) self.pool = nn.MaxPool2d(2, 2) self.conv2 = nn.Conv2d(6, 16, 5) self.fc1 = nn.Linear(16 * 5 * 5, 120) self.fc2 = nn.Linear(120, 84) self.fc3 = nn.Linear(84, 10) def forward(self, x): x = self.pool(F.relu(self.conv1(x))) x = self.pool(F.relu(self.conv2(x))) x = x.view(-1, 16 * 5 * 5) x = F.relu(self.fc1(x)) x = F.relu(self.fc2(x)) x = self.fc3(x) return x net = Net() print(net) optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9) # Collect all relevant information and build your dictionary. # Additional information EPOCH = 5 PATH = "model.pt" LOSS = 0.4 # 不光要保存模型的权重，还要保存优化器的状态量、Loss、Epoch torch.save({ 'epoch': EPOCH, 'model_state_dict': net.state_dict(), 'optimizer_state_dict': optimizer.state_dict(), 'loss': LOSS, }, PATH)Remember to first initialize the model and optimizer, then load the dictionary locally.model = Net() optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9) checkpoint = torch.load(PATH) model.load_state_dict(checkpoint['model_state_dict']) optimizer.load_state_dict(checkpoint['optimizer_state_dict']) epoch = checkpoint['epoch'] lo…

24、手写并验证向量内积实现PyTorch二维卷积
type: item
date: 2024-03-24T20:59:00+08:00
description: 该9步过程可以看作是一个矩阵跟另外一个矩阵的矩阵相乘将每步input拉直，例如左上角第一个深蓝色区域拉为332001312行向量，再将Kernel拉为$(012220012)^T$列向量相乘得到第一个最终结果的左上角第一个数值12.0So，上边九步运算可以视为行数为9的矩阵和列数为9的矩阵进行矩阵乘法，再将相乘的结果reshape为欲得到的结果Other Method实现一个长度为25的内积目前Kernel只是3×3的范围大小，若是将Kernel填充一下，eg.左上角第一幅图，Kernel只有9个数，但是我们可以脑补一下浅蓝色部分都填充为0，每一步都将Kernel填充浅蓝色部分为0，那么，原问题就变为25行向量和25列向量相乘。Coding希望把region_vector都放入到region_matrix中，再将region_matrix与Kernel_matrix相乘input = torch.randn(5, 5) # 卷积输入特征图 kernel = torch.randn(3, 3) # 卷积核 bias = torch.randn(1) # 卷积偏置项，默认输出通道数目＝1 # Func2 input flatten 版本 def matrix_mutiplication_for_conv2d_flatten(input, kernel, bias=0, stride=1, padding=0): if padding > 0: input = F.pad(input, (padding, padding, padding, padding)) # 左右上下都pad0 input_h, input_w = input.shape kernel_h, kernel_w = kernel.shape output_w = (math.floor((input_w - kernel_w) / stride) + 1) # 卷积输出的宽度 output_h = (math.floor((input_h - kernel_h) / stride) + 1) # 卷积输出的高度 output = torch.zeros(output_h, output_w) # 初始化输出矩阵 # numel()返回tensor中元素数目 # 上图所示一共9行9列 region_matrix = torch.zeros(output.numel(), kernel.numel()) # 定义region_matrix 存储所有展平后的特征区域 kernel_matrix = kernel.reshape((kernel.numel(), 1)) # 定义kernel_matrix 存储kernel的列向量形式 roll_index = 0 for i in range(0, input_h - kernel_h + 1, stride): # 对高度维度进行遍历 for j in range(0, input_w - kernel_w + 1, stride): # 对宽度维度进行遍历 region = input[i:i+kernel_h, j:j+kernel_w] # 取出被核滑动到的区域 3*3=9 region_vector = torch.flatten(region) # 将该区域展平 3*3=>1*9 region_matrix[roll_index] = region_vector # 每次放入一行1*9，9次将region_matrix全部铺满 roll_index += 1 # 下次再下一行赋值 output_matrix = region_matrix @ kernel_matrix # 两个矩阵相乘，就是对9次的input滑动窗口中的3*3展平后的1*9行向量✖kernel的列向量相乘9次加起来 output = output_matrix.reshape((output_h, output_w)) + bias return output # 矩阵运算实现卷积的结果 flatten版本 mat_mul_conv_flatten_output = matrix_mutiplication_for_conv2d_flatten(input, kernel, bias=bias, padding=1) print(mat_mul_conv_flatten_output.shape, "\n", mat_mul_conv_flatten_output) print(pytorch_api_conv_output.squeeze(0).squeeze(0).shape, "\n", pytorch_api_conv_outpu…

23、手写并验证滑动相乘实现PyTorch二维卷积
type: item
date: 2024-03-24T20:56:00+08:00
description: 蓝色的input_feature 5*5深蓝色小字部分kernel_size 3*3绿色部分out_feature 3*3stride = 1padding = 0channel = 1padding = 1stride = 2底部input_channels = 2顶端绿色为out_channels = 3kernels = 2*3 = 6（倒数第二行）input = input_feature_map # 卷积输入特征图 kernel = conv_layer.weight.data # 卷积核 input = torch.randn(5, 5) # 卷积输入特征图 kernel = torch.randn(3, 3) # 卷积核 bias = torch.randn(1) # 卷积偏置项，默认输出通道数目＝1 # Func1 用原始的矩阵运算来实现二维卷积,先不考虑batch_size和channel维度 def matrix_mutiplication_for_conv2d(input, kernel, bias=0, stride=1, padding=0): if padding > 0: input = F.pad(input, (padding, padding, padding, padding)) # 左右上下都pad0 input_h, input_w = input.shape kernel_h, kernel_w = kernel.shape output_w = (math.floor((input_w - kernel_w) / stride) + 1) # 卷积输出的宽度 output_h = (math.floor((input_h - kernel_h) / stride) + 1) # 卷积输出的高度 output = torch.zeros(output_h, output_w) # 初始化输出矩阵 for i in range(0, input_h - kernel_h + 1, stride): # 对高度维度进行遍历 for j in range(0, input_w - kernel_w + 1, stride): # 对宽度维度进行遍历 region = input[i:i+kernel_h, j:j+kernel_w] # 取出被核滑动到的区域 output[int(i / stride), int(j / stride)] = torch.sum(region * kernel) + bias # 点乘,并赋值给输出 return output # 矩阵运算实现卷积的结果 mat_mul_conv_output = matrix_mutiplication_for_conv2d(input, kernel, bias=bias, padding=1) print(mat_mul_conv_output.shape, "\n", mat_mul_conv_output) # 调用PyTorch API卷积的结果 pytorch_api_conv_output = F.conv2d(input.reshape((1,1,input.shape[0], input.shape[1])), \ kernel.reshape((1,1,kernel.shape[0], kernel.shape[1])),\ padding=1,\ bias=bias) print(pytorch_api_conv_output.squeeze(0).squeeze(0).shape, "\n", pytorch_api_conv_output.squeeze(0).squeeze(0)) # 验证成功，举证乘法实现的卷积欲pytorch api的结果一致(通道数为1) ''' torch.Size([5, 5]) tensor([[ 0.0254, 2.5571, 0.8080, -2.0241, 3.9600], [ 1.6969, 5.7820, -1.8596, 2.6106, 7.2310], [ 2.8070, 4.2363, -3.0085, 6.3041, 1.2186], [ 5.0747, -1.0790, 0.8183, 3.5965, -2.4651], [ 4.1119, -1.5446, 2.8745, 0.9343, 1.6732]]) torch.Size([5, 5]) tensor([[ 0.0254, 2.5571, 0.8080, -2.0241, 3.9600], [ 1.6969, 5.7820, -1.8596, 2.6106, 7.2310], [ 2.…