TensorFlow Graph Optimizations.pdf详细地阐述了TensorFlow

文件名称: TensorFlow Graph Optimizations.pdf

所属分类: 机器学习

开发工具:

文件大小: 1mb

下载次数: 0

上传时间: 2019-07-06

提供者: cheng*****

下载 (1mb)

不能下载？报告错误

详细说明：详细地阐述了TensorFlow上执行图优化算法，对于深入了解TensorFlow框架很有意义，为各种DAG执行引擎的设计提供技术指导！Open, standard software for general machine learning TTensorFlow Great for Deep Learning in particular http://tensorflow.org/ First released Nov 2015 and Apache 2.0 license https://github.com/tensorflow/tensorflow Powers many Google products TensorFlow Graph concepts TensorFlow(v1.x) programs generate a Data Flow(directed, multi-) Grapl o Device independent intermediate program representation o TensorFlow v2x uses a mix of imperative(Eager) execution mode and graphs Functions Graph nodes represent operations "Ops"(Add, MatMul, Conv2D,. Abstract device-, execution backend, and language independent AP o Implemented by Op Kernels written in C++, specialized on Graph edges represent data" flowing between ops o Tensors (ref-counted, n-dimensional array buffers in device memory o Control dependencies: A->B means A must finish before B can run o Resource handles to state(e.g. variables, input data pipelines Goog e Graph example: The Inception Architecture(2014) 国了围量量量圄冒圄晋福 Going Deeper with Convolutions Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru erhan vincent vanhoucke Andrew rabinovich Goog e Arxⅳv2014,cVPR2015 Output probabilities Graph example: The Transformer Softmax Add& norm Feed Forward Add norm Concat Add Norm Multi-Head Feed Attention Forward Scaled Dot-Product Attention h Add Norm Add norm Linear Linear Linear Masked Multi-Head Multi-Head Attention Attention K Q Positional Positional Encoding Encoding Attention Is All You Need (arXiv 2017) Output Embedding Embedding Ashish vaswani. noam shazeer Niki Parmar. Jakob uts Outputs Uszkoreit Llion Jones. Aidan n gomez. lukasz Kaiser (shifted right) Illia polosukhin Grappler Goog gle Grappler: Grappling with TF Graphs Grappler: Default graph optimization system in the TF runtime Re-writes graphs to improve out-of-the-box Tensor Flow performance o Provides a plugin infrastructure to register custom optimizers/rewriters Main goals o Automatically improve TF performance through graph simplifications high-level optimizations that benefit most target HW architectures (CPU/GPU/TPU/mobile etc.) Reduce device peak memory usage to enable larger models to run o Improve hardware utilization by optimizing the mapping of graph nodes to compute resources Provides cost models to drive optimization and help diagnose model performance Goog e Grappler: TensorFlow Context Python Swift Java C++ Grappler Graph XLA Compiler HLO TOCO TensorFlow. js TE runtime LLO LLVM IR executor FLIte TPU GPU/CPU Mobile Javascript GPU/CPU Goog e Embedded WebGl Why transformations at the graph level? Pros: Many optimizations can be easier to discover and express as high-level graph transformations Example: Matmul(Transpose( x), y)=> Matmul(x,y, transpose x=True) Graph is backend independent (tF runtime, XLA, TensorRT, TensorFlow js o Interoperable with Tensor Flow supported languages(protocol buffer format o Optimizations can be applied at runtime or offline using our standalone tool o Lots of existing models(TF Hub, Google production models) available for learning o Pragmatic: Helps the most existing Tensor Flow users get better "out-of-the-box"performance Cons: o Rewrites can be tricky to implement correctly, because of loosely defined graph semantics In-place ops, side-effects, control flow, control dependencies Protocol buffer dependence increases binary size o Currently requires extra graph format conversions in TF runtime Goog e

(系统自动生成,下载前可以参看下载内容)