Observations
HighLevelGraphs
How to get the HighLevelGraph:
-
res.data.daskorres.__dask_graph__()to get the graph in layers format. -
res.data.dask.visualize()orres.__dask_graph__().visualize()to get in graph format.
Internals:
-
HLG is a dict,
res.data.dask.__dict__. It has the nodes and keys of the task graphs. (which is basically the low level graph’s nodes) -
res.data.dask.dependenciesto get the layers, dependencies and key_dependencies. -
res.__dask_graph__().keysorres.data.dask.keyswould give the layers as keys. (res.data.dask.values also gives the same, i.e keys and values are same) -
On
res.data.dask.we can perform these operations [cull, merge, cull_layers, validate, get, copy, dependents, get_all_dependencies, visualize].
Notes:
type(dask.visualize(res))this gets the low level graph, but the type is an Image.type(res.data.dask)this gets the HLG, and the type is also HLG.
Fuse and HighLevelGraph
# from dask.distributed import Client
# client = Client(address = "tcp://192.168.0.100:8786")# clientimport xarray as xr
import dask.array as da
import numpy as np
# (a+b)^2 = a^2 + b^2 + 2ab
a = da.from_array([1, 2, 3, 4], chunks = (1))
b = da.from_array([5, 6, 7, 8], chunks = (1))
xr_a = xr.DataArray(a)
xr_b = xr.DataArray(b)
def add(arr1, arr2):
return arr1 + arr2
def sq(arr):
return arr ** 2
def mul(arr1, arr2):
return arr1 * arr2
a_sq = sq(xr_a)
b_sq = sq(xr_b)
add_ab = add(a_sq, b_sq) # a^2 +b^2
mul_ab = mul(xr_a, xr_b) # ab
res = add(add_ab, mul_ab)
# res.__dask_layers__() -> dk
# dask.visualize(res) -> low level graphs
# dask.optimize(a)
# a.npartitions # gives no.of chunks
# a.chunksize # gives chunk size
# res.__dask_keys__() -> dont know the real reason behind the keys
# compute
# dask delayed
# from_collections()
res.compute()html[theme=dark], html[data-theme=dark], body[data-theme=dark], body.vscode-dark { —xr-font-color0: rgba(255, 255, 255, 1); —xr-font-color2: rgba(255, 255, 255, 0.54); —xr-font-color3: rgba(255, 255, 255, 0.38); —xr-border-color: 1F1F1F; —xr-disabled-color: #515151; —xr-background-color: #111111; —xr-background-color-row-even: #111111; —xr-background-color-row-odd: #313131; }
.xr-wrap { display: block !important; min-width: 300px; max-width: 700px; }
.xr-text-repr-fallback { /* fallback to plain text repr when CSS is not injected (untrusted notebook) */ display: none; }
.xr-header { padding-top: 6px; padding-bottom: 6px; margin-bottom: 4px; border-bottom: solid 1px var(—xr-border-color); }
.xr-header > div, .xr-header > ul { display: inline; margin-top: 0; margin-bottom: 0; }
.xr-obj-type, .xr-array-name { margin-left: 2px; margin-right: 10px; }
.xr-obj-type { color: var(—xr-font-color2); }
.xr-sections { padding-left: 0 !important; display: grid; grid-template-columns: 150px auto auto 1fr 20px 20px; }
.xr-section-item { display: contents; }
.xr-section-item input { display: none; }
.xr-section-item input + label { color: var(—xr-disabled-color); }
.xr-section-item input:enabled + label { cursor: pointer; color: var(—xr-font-color2); }
.xr-section-item input:enabled + label:hover { color: var(—xr-font-color0); }
.xr-section-summary { grid-column: 1; color: var(—xr-font-color2); font-weight: 500; }
.xr-section-summary > span { display: inline-block; padding-left: 0.5em; }
.xr-section-summary-in:disabled + label { color: var(—xr-font-color2); }
.xr-section-summary-in + label:before { display: inline-block; content: ’►’; font-size: 11px; width: 15px; text-align: center; }
.xr-section-summary-in:disabled + label:before { color: var(—xr-disabled-color); }
.xr-section-summary-in:checked + label:before { content: ’▼’; }
.xr-section-summary-in:checked + label > span { display: none; }
.xr-section-summary, .xr-section-inline-details { padding-top: 4px; padding-bottom: 4px; }
.xr-section-inline-details { grid-column: 2 / -1; }
.xr-section-details { display: none; grid-column: 1 / -1; margin-bottom: 5px; }
.xr-section-summary-in:checked ~ .xr-section-details { display: contents; }
.xr-array-wrap { grid-column: 1 / -1; display: grid; grid-template-columns: 20px auto; }
.xr-array-wrap > label { grid-column: 1; vertical-align: top; }
.xr-preview { color: var(—xr-font-color3); }
.xr-array-preview, .xr-array-data { padding: 0 5px !important; grid-column: 2; }
.xr-array-data, .xr-array-in:checked ~ .xr-array-preview { display: none; }
.xr-array-in:checked ~ .xr-array-data, .xr-array-preview { display: inline-block; }
.xr-dim-list { display: inline-block !important; list-style: none; padding: 0 !important; margin: 0; }
.xr-dim-list li { display: inline-block; padding: 0; margin: 0; }
.xr-dim-list:before { content: ’(’; }
.xr-dim-list:after { content: ’)’; }
.xr-dim-list li:not(:last-child):after { content: ’,’; padding-right: 5px; }
.xr-has-index { font-weight: bold; }
.xr-var-list, .xr-var-item { display: contents; }
.xr-var-item > div, .xr-var-item label, .xr-var-item > .xr-var-name span { background-color: var(—xr-background-color-row-even); margin-bottom: 0; }
.xr-var-item > .xr-var-name:hover span { padding-right: 5px; }
.xr-var-list > li:nth-child(odd) > div, .xr-var-list > li:nth-child(odd) > label, .xr-var-list > li:nth-child(odd) > .xr-var-name span { background-color: var(—xr-background-color-row-odd); }
.xr-var-name { grid-column: 1; }
.xr-var-dims { grid-column: 2; }
.xr-var-dtype { grid-column: 3; text-align: right; color: var(—xr-font-color2); }
.xr-var-preview { grid-column: 4; }
.xr-index-preview { grid-column: 2 / 5; color: var(—xr-font-color2); }
.xr-var-name, .xr-var-dims, .xr-var-dtype, .xr-preview, .xr-attrs dt { white-space: nowrap; overflow: hidden; text-overflow: ellipsis; padding-right: 10px; }
.xr-var-name:hover, .xr-var-dims:hover, .xr-var-dtype:hover, .xr-attrs dt:hover { overflow: visible; width: auto; z-index: 1; }
.xr-var-attrs, .xr-var-data, .xr-index-data { display: none; background-color: var(—xr-background-color) !important; padding-bottom: 5px !important; }
.xr-var-attrs-in:checked ~ .xr-var-attrs, .xr-var-data-in:checked ~ .xr-var-data, .xr-index-data-in:checked ~ .xr-index-data { display: block; }
.xr-var-data > table { float: right; }
.xr-var-name span, .xr-var-data, .xr-index-name div, .xr-index-data, .xr-attrs { padding-left: 25px !important; }
.xr-attrs, .xr-var-attrs, .xr-var-data, .xr-index-data { grid-column: 1 / -1; }
dl.xr-attrs { padding: 0; margin: 0; display: grid; grid-template-columns: 125px auto; }
.xr-attrs dt, .xr-attrs dd { padding: 0; margin: 0; float: left; padding-right: 10px; width: auto; }
.xr-attrs dt { font-weight: normal; grid-column: 1; }
.xr-attrs dt:hover span { display: inline-block; background: var(—xr-background-color); padding-right: 10px; }
.xr-attrs dd { grid-column: 2; white-space: pre-wrap; word-break: break-all; }
.xr-icon-database, .xr-icon-file-text2, .xr-no-icon { display: inline-block; vertical-align: middle; width: 1em; height: 1.5em !important; stroke-width: 0; stroke: currentColor; fill: currentColor; }
<xarray.DataArray (dim_0: 4)> Size: 32B array([ 31, 52, 79, 112]) Dimensions without coordinates: dim_0
res.data.dask.visualize()
import dask
graph = res.data.dask
hlg_keys = res.data.dask.keysfrom dask.threaded import get
from dask.optimization import fuse, fuse_linear
outputs = "add-ffe880f167c9131320013be9069dfca1"
graph2, dep = fuse(dict(graph), fuse_subgraphs=True)keys = graph2.keys()
print(keys)
get(graph2, ('add-ffe880f167c9131320013be9069dfca1', 0))dict_keys([('add-ffe880f167c9131320013be9069dfca1', 0), ('add-ffe880f167c9131320013be9069dfca1', 1), ('add-ffe880f167c9131320013be9069dfca1', 2), ('add-ffe880f167c9131320013be9069dfca1', 3), ('add-d8368b0c413b7b1ff3e4545d5d219543', 0), ('add-d8368b0c413b7b1ff3e4545d5d219543', 1), ('add-d8368b0c413b7b1ff3e4545d5d219543', 2), ('add-d8368b0c413b7b1ff3e4545d5d219543', 3), ('array-93fac394d6028ca1841a0aaaf012fdfd', 0), ('array-93fac394d6028ca1841a0aaaf012fdfd', 1), ('array-93fac394d6028ca1841a0aaaf012fdfd', 2), ('array-93fac394d6028ca1841a0aaaf012fdfd', 3), ('pow-f98325ba6ded8b13c46607d966ba51d1', 0), ('pow-f98325ba6ded8b13c46607d966ba51d1', 1), ('pow-f98325ba6ded8b13c46607d966ba51d1', 2), ('pow-f98325ba6ded8b13c46607d966ba51d1', 3), ('array-e0289449abd23dc644f0fd074cd695dc', 0), ('array-e0289449abd23dc644f0fd074cd695dc', 1), ('array-e0289449abd23dc644f0fd074cd695dc', 2), ('array-e0289449abd23dc644f0fd074cd695dc', 3), ('pow-5022bc0a56d6e28f1fd172a750ba8cff', 0), ('pow-5022bc0a56d6e28f1fd172a750ba8cff', 1), ('pow-5022bc0a56d6e28f1fd172a750ba8cff', 2), ('pow-5022bc0a56d6e28f1fd172a750ba8cff', 3), ('mul-8e9e32e19f9c6e612b3800864d8cc810', 0), ('mul-8e9e32e19f9c6e612b3800864d8cc810', 1), ('mul-8e9e32e19f9c6e612b3800864d8cc810', 2), ('mul-8e9e32e19f9c6e612b3800864d8cc810', 3)])
array([31])
Call delayed on HLG
from dask.delayed import Delayed, delayed
root_key = "add-ffe880f167c9131320013be9069dfca1"
g1 = graph
result = delayed(graph)
result.compute()HighLevelGraph
HighLevelGraph with 7 layers and 28 keys from all layers.
<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">
<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>
</svg>
<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer1: array</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
array-e0289449abd23dc644f0fd074cd695dc
</p>
<table>
<tr>
<td>
<table>
<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">MaterializedLayer</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">4</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">shape</th>
<td style="text-align: left;">(4,)</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">dtype</th>
<td style="text-align: left;">int64</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">chunksize</th>
<td style="text-align: left;">(1,)</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">type</th>
<td style="text-align: left;">dask.array.core.Array</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">chunk_type</th>
<td style="text-align: left;">numpy.ndarray</td>
</tr>
</table>
</td>
<td>
<svg width="250" height="121" style="stroke:rgb(0,0,0);stroke-width:1" >
</details>
<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">
<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>
</svg>
<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer2: pow</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
pow-5022bc0a56d6e28f1fd172a750ba8cff
</p>
<table>
<tr>
<td>
<table>
<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">Blockwise</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">4</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">shape</th>
<td style="text-align: left;">(4,)</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">dtype</th>
<td style="text-align: left;">int64</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">chunksize</th>
<td style="text-align: left;">(1,)</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">type</th>
<td style="text-align: left;">dask.array.core.Array</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">chunk_type</th>
<td style="text-align: left;">numpy.ndarray</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;"> depends on </th>
<td style="text-align: left;">array-e0289449abd23dc644f0fd074cd695dc</td>
</tr>
</table>
</td>
<td>
<svg width="250" height="121" style="stroke:rgb(0,0,0);stroke-width:1" >
</details>
<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">
<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>
</svg>
<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer3: array</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
array-93fac394d6028ca1841a0aaaf012fdfd
</p>
<table>
<tr>
<td>
<table>
<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">MaterializedLayer</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">4</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">shape</th>
<td style="text-align: left;">(4,)</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">dtype</th>
<td style="text-align: left;">int64</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">chunksize</th>
<td style="text-align: left;">(1,)</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">type</th>
<td style="text-align: left;">dask.array.core.Array</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">chunk_type</th>
<td style="text-align: left;">numpy.ndarray</td>
</tr>
</table>
</td>
<td>
<svg width="250" height="121" style="stroke:rgb(0,0,0);stroke-width:1" >
</details>
<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">
<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>
</svg>
<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer4: mul</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
mul-8e9e32e19f9c6e612b3800864d8cc810
</p>
<table>
<tr>
<td>
<table>
<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">Blockwise</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">4</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">shape</th>
<td style="text-align: left;">(4,)</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">dtype</th>
<td style="text-align: left;">int64</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">chunksize</th>
<td style="text-align: left;">(1,)</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">type</th>
<td style="text-align: left;">dask.array.core.Array</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">chunk_type</th>
<td style="text-align: left;">numpy.ndarray</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;"> depends on </th>
<td style="text-align: left;">array-93fac394d6028ca1841a0aaaf012fdfd</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;"></th>
<td style="text-align: left;">array-e0289449abd23dc644f0fd074cd695dc</td>
</tr>
</table>
</td>
<td>
<svg width="250" height="121" style="stroke:rgb(0,0,0);stroke-width:1" >
</details>
<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">
<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>
</svg>
<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer5: pow</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
pow-f98325ba6ded8b13c46607d966ba51d1
</p>
<table>
<tr>
<td>
<table>
<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">Blockwise</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">4</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">shape</th>
<td style="text-align: left;">(4,)</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">dtype</th>
<td style="text-align: left;">int64</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">chunksize</th>
<td style="text-align: left;">(1,)</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">type</th>
<td style="text-align: left;">dask.array.core.Array</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">chunk_type</th>
<td style="text-align: left;">numpy.ndarray</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;"> depends on </th>
<td style="text-align: left;">array-93fac394d6028ca1841a0aaaf012fdfd</td>
</tr>
</table>
</td>
<td>
<svg width="250" height="121" style="stroke:rgb(0,0,0);stroke-width:1" >
</details>
<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">
<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>
</svg>
<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer6: add</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
add-d8368b0c413b7b1ff3e4545d5d219543
</p>
<table>
<tr>
<td>
<table>
<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">Blockwise</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">4</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">shape</th>
<td style="text-align: left;">(4,)</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">dtype</th>
<td style="text-align: left;">int64</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">chunksize</th>
<td style="text-align: left;">(1,)</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">type</th>
<td style="text-align: left;">dask.array.core.Array</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">chunk_type</th>
<td style="text-align: left;">numpy.ndarray</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;"> depends on </th>
<td style="text-align: left;">pow-5022bc0a56d6e28f1fd172a750ba8cff</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;"></th>
<td style="text-align: left;">pow-f98325ba6ded8b13c46607d966ba51d1</td>
</tr>
</table>
</td>
<td>
<svg width="250" height="121" style="stroke:rgb(0,0,0);stroke-width:1" >
</details>
<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">
<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>
</svg>
<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer7: add</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
add-ffe880f167c9131320013be9069dfca1
</p>
<table>
<tr>
<td>
<table>
<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">Blockwise</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">4</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">shape</th>
<td style="text-align: left;">(4,)</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">dtype</th>
<td style="text-align: left;">int64</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">chunksize</th>
<td style="text-align: left;">(1,)</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">type</th>
<td style="text-align: left;">dask.array.core.Array</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">chunk_type</th>
<td style="text-align: left;">numpy.ndarray</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;"> depends on </th>
<td style="text-align: left;">add-d8368b0c413b7b1ff3e4545d5d219543</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;"></th>
<td style="text-align: left;">mul-8e9e32e19f9c6e612b3800864d8cc810</td>
</tr>
</table>
</td>
<td>
<svg width="250" height="121" style="stroke:rgb(0,0,0);stroke-width:1" >
</details>
</div>
</div>
# https://github.com/ska-sa/katsdpcal/blob/e13c3442ae7be050a7455bc384116aae8e082895/katsdpcal/inplace.py#L250
# Returns equivalent dask collections that all share the same merged and
# optimized underlying graph. This can be useful if converting multiple
# collections to delayed objects, or to manually apply the optimizations at
# strategic points.
# https://github.com/dask/dask/blob/aafde77e5c7f5220775635a0035c7ae64310ed3a/dask/base.py#L588
# def repack(results):
# dsk = repack_dsk.copy()
# dsk[collections_token] = quote(results)
# return simple_get(dsk, out)
# Quote: Ensure that this value remains this value in a dask graph https://github.com/dask/dask/blob/main/dask/core.py#L515
dask delayed and fuse
import xarray as xr
import dask.array as da
import numpy as np
# (a+b)^2 = a^2 + b^2 + 2ab
a = da.from_array([1, 2, 3, 4], chunks = (1))
b = da.from_array([5, 6, 7, 8], chunks = (1))
xr_a = xr.DataArray(a)
xr_b = xr.DataArray(b)
@dask.delayed
def add_d(arr1, arr2):
return arr1 + arr2
@dask.delayed
def sq_d(arr):
return arr ** 2
@dask.delayed
def mul_d(arr1, arr2):
return arr1 * arr2
a_sq = sq_d(xr_a)
b_sq = sq_d(xr_b)
add_ab = add_d(a_sq, b_sq) # a^2 +b^2
mul_ab = mul_d(xr_a, xr_b) # ab
res_d = add(add_ab, mul_ab)
delayed_hgl = res_d.data.dask
delayed_hgl.visualize()
dict(delayed_hgl){'add-a362066893b349d99b2b19a5d658fccb': (<function _operator.add(a, b, /)>,
'add_d-a0be0541-c05e-46af-b8c5-df5fb17eea90',
'mul_d-f2fc9779-b3f6-4bec-8822-f5f6fe8041e0'),
'add_d-a0be0541-c05e-46af-b8c5-df5fb17eea90': (<function __main__.add_d(arr1, arr2)>,
'sq_d-96134174-5594-4c11-bb93-b045188f8a65',
'sq_d-57df3f05-4ad2-41b4-bf02-4f82ddcaa741'),
('array-93fac394d6028ca1841a0aaaf012fdfd', 0): array([1]),
('array-93fac394d6028ca1841a0aaaf012fdfd', 1): array([2]),
('array-93fac394d6028ca1841a0aaaf012fdfd', 2): array([3]),
('array-93fac394d6028ca1841a0aaaf012fdfd', 3): array([4]),
'finalize-71ed3afe-61b1-46e5-8a49-2c36a9e8826f': (<bound method DataArray._dask_finalize of <class 'xarray.core.dataarray.DataArray'>>,
[[('array-93fac394d6028ca1841a0aaaf012fdfd', 0),
('array-93fac394d6028ca1841a0aaaf012fdfd', 1),
('array-93fac394d6028ca1841a0aaaf012fdfd', 2),
('array-93fac394d6028ca1841a0aaaf012fdfd', 3)]],
'array-93fac394d6028ca1841a0aaaf012fdfd',
<bound method Dataset._dask_postcompute of <xarray.Dataset> Size: 32B
Dimensions: (dim_0: 4)
Dimensions without coordinates: dim_0
Data variables:
<this-array> (dim_0) int64 32B dask.array<chunksize=(1,), meta=np.ndarray>>),
'sq_d-96134174-5594-4c11-bb93-b045188f8a65': (<function __main__.sq_d(arr)>,
'finalize-71ed3afe-61b1-46e5-8a49-2c36a9e8826f'),
('array-e0289449abd23dc644f0fd074cd695dc', 0): array([5]),
('array-e0289449abd23dc644f0fd074cd695dc', 1): array([6]),
('array-e0289449abd23dc644f0fd074cd695dc', 2): array([7]),
('array-e0289449abd23dc644f0fd074cd695dc', 3): array([8]),
'finalize-89a5a21d-b67f-47b9-969d-bf82c9f632d6': (<bound method DataArray._dask_finalize of <class 'xarray.core.dataarray.DataArray'>>,
[[('array-e0289449abd23dc644f0fd074cd695dc', 0),
('array-e0289449abd23dc644f0fd074cd695dc', 1),
('array-e0289449abd23dc644f0fd074cd695dc', 2),
('array-e0289449abd23dc644f0fd074cd695dc', 3)]],
'array-e0289449abd23dc644f0fd074cd695dc',
<bound method Dataset._dask_postcompute of <xarray.Dataset> Size: 32B
Dimensions: (dim_0: 4)
Dimensions without coordinates: dim_0
Data variables:
<this-array> (dim_0) int64 32B dask.array<chunksize=(1,), meta=np.ndarray>>),
'sq_d-57df3f05-4ad2-41b4-bf02-4f82ddcaa741': (<function __main__.sq_d(arr)>,
'finalize-89a5a21d-b67f-47b9-969d-bf82c9f632d6'),
'mul_d-f2fc9779-b3f6-4bec-8822-f5f6fe8041e0': (<function __main__.mul_d(arr1, arr2)>,
'finalize-8e4a37e5-68d7-4d57-908c-05611cb218a2',
'finalize-abb81368-05d0-48fd-bc81-4da3d383a026'),
'finalize-8e4a37e5-68d7-4d57-908c-05611cb218a2': (<bound method DataArray._dask_finalize of <class 'xarray.core.dataarray.DataArray'>>,
[[('array-93fac394d6028ca1841a0aaaf012fdfd', 0),
('array-93fac394d6028ca1841a0aaaf012fdfd', 1),
('array-93fac394d6028ca1841a0aaaf012fdfd', 2),
('array-93fac394d6028ca1841a0aaaf012fdfd', 3)]],
'array-93fac394d6028ca1841a0aaaf012fdfd',
<bound method Dataset._dask_postcompute of <xarray.Dataset> Size: 32B
Dimensions: (dim_0: 4)
Dimensions without coordinates: dim_0
Data variables:
<this-array> (dim_0) int64 32B dask.array<chunksize=(1,), meta=np.ndarray>>),
'finalize-abb81368-05d0-48fd-bc81-4da3d383a026': (<bound method DataArray._dask_finalize of <class 'xarray.core.dataarray.DataArray'>>,
[[('array-e0289449abd23dc644f0fd074cd695dc', 0),
('array-e0289449abd23dc644f0fd074cd695dc', 1),
('array-e0289449abd23dc644f0fd074cd695dc', 2),
('array-e0289449abd23dc644f0fd074cd695dc', 3)]],
'array-e0289449abd23dc644f0fd074cd695dc',
<bound method Dataset._dask_postcompute of <xarray.Dataset> Size: 32B
Dimensions: (dim_0: 4)
Dimensions without coordinates: dim_0
Data variables:
<this-array> (dim_0) int64 32B dask.array<chunksize=(1,), meta=np.ndarray>>),
'getattr-d83101a6b9634af52ae2ed30c7d8dba0': (<function getattr>,
'add-a362066893b349d99b2b19a5d658fccb',
'data')}
client.get(dict(delayed_hgl), 'add-a362066893b349d99b2b19a5d658fccb')
# get(dict(delayed_hgl), 'add-a362066893b349d99b2b19a5d658fccb')html[theme=dark], html[data-theme=dark], body[data-theme=dark], body.vscode-dark { —xr-font-color0: rgba(255, 255, 255, 1); —xr-font-color2: rgba(255, 255, 255, 0.54); —xr-font-color3: rgba(255, 255, 255, 0.38); —xr-border-color: 1F1F1F; —xr-disabled-color: #515151; —xr-background-color: #111111; —xr-background-color-row-even: #111111; —xr-background-color-row-odd: #313131; }
.xr-wrap { display: block !important; min-width: 300px; max-width: 700px; }
.xr-text-repr-fallback { /* fallback to plain text repr when CSS is not injected (untrusted notebook) */ display: none; }
.xr-header { padding-top: 6px; padding-bottom: 6px; margin-bottom: 4px; border-bottom: solid 1px var(—xr-border-color); }
.xr-header > div, .xr-header > ul { display: inline; margin-top: 0; margin-bottom: 0; }
.xr-obj-type, .xr-array-name { margin-left: 2px; margin-right: 10px; }
.xr-obj-type { color: var(—xr-font-color2); }
.xr-sections { padding-left: 0 !important; display: grid; grid-template-columns: 150px auto auto 1fr 20px 20px; }
.xr-section-item { display: contents; }
.xr-section-item input { display: none; }
.xr-section-item input + label { color: var(—xr-disabled-color); }
.xr-section-item input:enabled + label { cursor: pointer; color: var(—xr-font-color2); }
.xr-section-item input:enabled + label:hover { color: var(—xr-font-color0); }
.xr-section-summary { grid-column: 1; color: var(—xr-font-color2); font-weight: 500; }
.xr-section-summary > span { display: inline-block; padding-left: 0.5em; }
.xr-section-summary-in:disabled + label { color: var(—xr-font-color2); }
.xr-section-summary-in + label:before { display: inline-block; content: ’►’; font-size: 11px; width: 15px; text-align: center; }
.xr-section-summary-in:disabled + label:before { color: var(—xr-disabled-color); }
.xr-section-summary-in:checked + label:before { content: ’▼’; }
.xr-section-summary-in:checked + label > span { display: none; }
.xr-section-summary, .xr-section-inline-details { padding-top: 4px; padding-bottom: 4px; }
.xr-section-inline-details { grid-column: 2 / -1; }
.xr-section-details { display: none; grid-column: 1 / -1; margin-bottom: 5px; }
.xr-section-summary-in:checked ~ .xr-section-details { display: contents; }
.xr-array-wrap { grid-column: 1 / -1; display: grid; grid-template-columns: 20px auto; }
.xr-array-wrap > label { grid-column: 1; vertical-align: top; }
.xr-preview { color: var(—xr-font-color3); }
.xr-array-preview, .xr-array-data { padding: 0 5px !important; grid-column: 2; }
.xr-array-data, .xr-array-in:checked ~ .xr-array-preview { display: none; }
.xr-array-in:checked ~ .xr-array-data, .xr-array-preview { display: inline-block; }
.xr-dim-list { display: inline-block !important; list-style: none; padding: 0 !important; margin: 0; }
.xr-dim-list li { display: inline-block; padding: 0; margin: 0; }
.xr-dim-list:before { content: ’(’; }
.xr-dim-list:after { content: ’)’; }
.xr-dim-list li:not(:last-child):after { content: ’,’; padding-right: 5px; }
.xr-has-index { font-weight: bold; }
.xr-var-list, .xr-var-item { display: contents; }
.xr-var-item > div, .xr-var-item label, .xr-var-item > .xr-var-name span { background-color: var(—xr-background-color-row-even); margin-bottom: 0; }
.xr-var-item > .xr-var-name:hover span { padding-right: 5px; }
.xr-var-list > li:nth-child(odd) > div, .xr-var-list > li:nth-child(odd) > label, .xr-var-list > li:nth-child(odd) > .xr-var-name span { background-color: var(—xr-background-color-row-odd); }
.xr-var-name { grid-column: 1; }
.xr-var-dims { grid-column: 2; }
.xr-var-dtype { grid-column: 3; text-align: right; color: var(—xr-font-color2); }
.xr-var-preview { grid-column: 4; }
.xr-index-preview { grid-column: 2 / 5; color: var(—xr-font-color2); }
.xr-var-name, .xr-var-dims, .xr-var-dtype, .xr-preview, .xr-attrs dt { white-space: nowrap; overflow: hidden; text-overflow: ellipsis; padding-right: 10px; }
.xr-var-name:hover, .xr-var-dims:hover, .xr-var-dtype:hover, .xr-attrs dt:hover { overflow: visible; width: auto; z-index: 1; }
.xr-var-attrs, .xr-var-data, .xr-index-data { display: none; background-color: var(—xr-background-color) !important; padding-bottom: 5px !important; }
.xr-var-attrs-in:checked ~ .xr-var-attrs, .xr-var-data-in:checked ~ .xr-var-data, .xr-index-data-in:checked ~ .xr-index-data { display: block; }
.xr-var-data > table { float: right; }
.xr-var-name span, .xr-var-data, .xr-index-name div, .xr-index-data, .xr-attrs { padding-left: 25px !important; }
.xr-attrs, .xr-var-attrs, .xr-var-data, .xr-index-data { grid-column: 1 / -1; }
dl.xr-attrs { padding: 0; margin: 0; display: grid; grid-template-columns: 125px auto; }
.xr-attrs dt, .xr-attrs dd { padding: 0; margin: 0; float: left; padding-right: 10px; width: auto; }
.xr-attrs dt { font-weight: normal; grid-column: 1; }
.xr-attrs dt:hover span { display: inline-block; background: var(—xr-background-color); padding-right: 10px; }
.xr-attrs dd { grid-column: 2; white-space: pre-wrap; word-break: break-all; }
.xr-icon-database, .xr-icon-file-text2, .xr-no-icon { display: inline-block; vertical-align: middle; width: 1em; height: 1.5em !important; stroke-width: 0; stroke: currentColor; fill: currentColor; }
<xarray.DataArray (dim_0: 4)> Size: 32B array([ 31, 52, 79, 112]) Dimensions without coordinates: dim_0
delayed_hglHighLevelGraph
HighLevelGraph with 12 layers and 18 keys from all layers.
<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">
<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>
</svg>
<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer1: array</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
array-e0289449abd23dc644f0fd074cd695dc
</p>
<table>
<tr>
<td>
<table>
<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">MaterializedLayer</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">4</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">shape</th>
<td style="text-align: left;">(4,)</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">dtype</th>
<td style="text-align: left;">int64</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">chunksize</th>
<td style="text-align: left;">(1,)</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">type</th>
<td style="text-align: left;">dask.array.core.Array</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">chunk_type</th>
<td style="text-align: left;">numpy.ndarray</td>
</tr>
</table>
</td>
<td>
<svg width="250" height="121" style="stroke:rgb(0,0,0);stroke-width:1" >
</details>
<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">
<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>
</svg>
<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer2: finalize</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
finalize-abb81368-05d0-48fd-bc81-4da3d383a026
</p>
<table>
<tr>
<td>
<table>
<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">MaterializedLayer</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">1</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;"> depends on </th>
<td style="text-align: left;">array-e0289449abd23dc644f0fd074cd695dc</td>
</tr>
</table>
</td>
<td>
</td>
</tr>
</table>
</details>
<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">
<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>
</svg>
<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer3: finalize</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
finalize-89a5a21d-b67f-47b9-969d-bf82c9f632d6
</p>
<table>
<tr>
<td>
<table>
<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">MaterializedLayer</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">1</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;"> depends on </th>
<td style="text-align: left;">array-e0289449abd23dc644f0fd074cd695dc</td>
</tr>
</table>
</td>
<td>
</td>
</tr>
</table>
</details>
<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">
<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>
</svg>
<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer4: sq_d</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
sq_d-57df3f05-4ad2-41b4-bf02-4f82ddcaa741
</p>
<table>
<tr>
<td>
<table>
<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">MaterializedLayer</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">1</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;"> depends on </th>
<td style="text-align: left;">finalize-89a5a21d-b67f-47b9-969d-bf82c9f632d6</td>
</tr>
</table>
</td>
<td>
</td>
</tr>
</table>
</details>
<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">
<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>
</svg>
<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer5: array</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
array-93fac394d6028ca1841a0aaaf012fdfd
</p>
<table>
<tr>
<td>
<table>
<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">MaterializedLayer</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">4</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">shape</th>
<td style="text-align: left;">(4,)</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">dtype</th>
<td style="text-align: left;">int64</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">chunksize</th>
<td style="text-align: left;">(1,)</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">type</th>
<td style="text-align: left;">dask.array.core.Array</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">chunk_type</th>
<td style="text-align: left;">numpy.ndarray</td>
</tr>
</table>
</td>
<td>
<svg width="250" height="121" style="stroke:rgb(0,0,0);stroke-width:1" >
</details>
<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">
<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>
</svg>
<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer6: finalize</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
finalize-8e4a37e5-68d7-4d57-908c-05611cb218a2
</p>
<table>
<tr>
<td>
<table>
<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">MaterializedLayer</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">1</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;"> depends on </th>
<td style="text-align: left;">array-93fac394d6028ca1841a0aaaf012fdfd</td>
</tr>
</table>
</td>
<td>
</td>
</tr>
</table>
</details>
<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">
<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>
</svg>
<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer7: mul_d</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
mul_d-f2fc9779-b3f6-4bec-8822-f5f6fe8041e0
</p>
<table>
<tr>
<td>
<table>
<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">MaterializedLayer</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">1</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;"> depends on </th>
<td style="text-align: left;">finalize-8e4a37e5-68d7-4d57-908c-05611cb218a2</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;"></th>
<td style="text-align: left;">finalize-abb81368-05d0-48fd-bc81-4da3d383a026</td>
</tr>
</table>
</td>
<td>
</td>
</tr>
</table>
</details>
<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">
<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>
</svg>
<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer8: finalize</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
finalize-71ed3afe-61b1-46e5-8a49-2c36a9e8826f
</p>
<table>
<tr>
<td>
<table>
<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">MaterializedLayer</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">1</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;"> depends on </th>
<td style="text-align: left;">array-93fac394d6028ca1841a0aaaf012fdfd</td>
</tr>
</table>
</td>
<td>
</td>
</tr>
</table>
</details>
<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">
<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>
</svg>
<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer9: sq_d</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
sq_d-96134174-5594-4c11-bb93-b045188f8a65
</p>
<table>
<tr>
<td>
<table>
<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">MaterializedLayer</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">1</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;"> depends on </th>
<td style="text-align: left;">finalize-71ed3afe-61b1-46e5-8a49-2c36a9e8826f</td>
</tr>
</table>
</td>
<td>
</td>
</tr>
</table>
</details>
<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">
<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>
</svg>
<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer10: add_d</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
add_d-a0be0541-c05e-46af-b8c5-df5fb17eea90
</p>
<table>
<tr>
<td>
<table>
<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">MaterializedLayer</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">1</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;"> depends on </th>
<td style="text-align: left;">sq_d-57df3f05-4ad2-41b4-bf02-4f82ddcaa741</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;"></th>
<td style="text-align: left;">sq_d-96134174-5594-4c11-bb93-b045188f8a65</td>
</tr>
</table>
</td>
<td>
</td>
</tr>
</table>
</details>
<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">
<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>
</svg>
<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer11: add</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
add-a362066893b349d99b2b19a5d658fccb
</p>
<table>
<tr>
<td>
<table>
<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">MaterializedLayer</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">1</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;"> depends on </th>
<td style="text-align: left;">add_d-a0be0541-c05e-46af-b8c5-df5fb17eea90</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;"></th>
<td style="text-align: left;">mul_d-f2fc9779-b3f6-4bec-8822-f5f6fe8041e0</td>
</tr>
</table>
</td>
<td>
</td>
</tr>
</table>
</details>
<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">
<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>
</svg>
<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer12: getattr</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
getattr-d83101a6b9634af52ae2ed30c7d8dba0
</p>
<table>
<tr>
<td>
<table>
<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">MaterializedLayer</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">1</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;"> depends on </th>
<td style="text-align: left;">add-a362066893b349d99b2b19a5d658fccb</td>
</tr>
</table>
</td>
<td>
</td>
</tr>
</table>
</details>
</div>
</div>
delayed_hgl.layers{'add-a362066893b349d99b2b19a5d658fccb': <dask.highlevelgraph.MaterializedLayer at 0x16447ceb0>,
'add_d-a0be0541-c05e-46af-b8c5-df5fb17eea90': <dask.highlevelgraph.MaterializedLayer at 0x16447cc10>,
'array-93fac394d6028ca1841a0aaaf012fdfd': <dask.highlevelgraph.MaterializedLayer at 0x164d21930>,
'finalize-71ed3afe-61b1-46e5-8a49-2c36a9e8826f': <dask.highlevelgraph.MaterializedLayer at 0x164d8c8e0>,
'sq_d-96134174-5594-4c11-bb93-b045188f8a65': <dask.highlevelgraph.MaterializedLayer at 0x164d22920>,
'array-e0289449abd23dc644f0fd074cd695dc': <dask.highlevelgraph.MaterializedLayer at 0x164d8c1f0>,
'finalize-89a5a21d-b67f-47b9-969d-bf82c9f632d6': <dask.highlevelgraph.MaterializedLayer at 0x164d8ca00>,
'sq_d-57df3f05-4ad2-41b4-bf02-4f82ddcaa741': <dask.highlevelgraph.MaterializedLayer at 0x108e85c30>,
'mul_d-f2fc9779-b3f6-4bec-8822-f5f6fe8041e0': <dask.highlevelgraph.MaterializedLayer at 0x16447d2a0>,
'finalize-8e4a37e5-68d7-4d57-908c-05611cb218a2': <dask.highlevelgraph.MaterializedLayer at 0x164d8dd20>,
'finalize-abb81368-05d0-48fd-bc81-4da3d383a026': <dask.highlevelgraph.MaterializedLayer at 0x164d8dd50>,
'getattr-d83101a6b9634af52ae2ed30c7d8dba0': <dask.highlevelgraph.MaterializedLayer at 0x16447d000>}
from dask.optimization import fuse, fuse_linear
from dask.highlevelgraph import HighLevelGraph
# print(delayed_hgl.layers)
sel_layers = delayed_hgl.layers
merged_hgl = delayed_hgl.merge(sel_layers)
# print(dict(merged_hgl))
get(dict(merged_hgl),'add-a362066893b349d99b2b19a5d658fccb' )
# delayed_hgl.__dir__()
# help(HighLevelGraph.validate)
from dask.distributed import Client
client = Client("tcp://192.168.0.100:8786")
# help(dask.distributed.get)
# get(dict(merged_hgl) , 'add-a362066893b349d99b2b19a5d658fccb' )
# print(delayed_hgl.validate())
dict(merged_hgl){'add-a362066893b349d99b2b19a5d658fccb': <dask.highlevelgraph.MaterializedLayer at 0x16447ceb0>,
'add_d-a0be0541-c05e-46af-b8c5-df5fb17eea90': <dask.highlevelgraph.MaterializedLayer at 0x16447cc10>,
'array-93fac394d6028ca1841a0aaaf012fdfd': <dask.highlevelgraph.MaterializedLayer at 0x164d21930>,
'finalize-71ed3afe-61b1-46e5-8a49-2c36a9e8826f': <dask.highlevelgraph.MaterializedLayer at 0x164d8c8e0>,
'sq_d-96134174-5594-4c11-bb93-b045188f8a65': <dask.highlevelgraph.MaterializedLayer at 0x164d22920>,
'array-e0289449abd23dc644f0fd074cd695dc': <dask.highlevelgraph.MaterializedLayer at 0x164d8c1f0>,
'finalize-89a5a21d-b67f-47b9-969d-bf82c9f632d6': <dask.highlevelgraph.MaterializedLayer at 0x164d8ca00>,
'sq_d-57df3f05-4ad2-41b4-bf02-4f82ddcaa741': <dask.highlevelgraph.MaterializedLayer at 0x108e85c30>,
'mul_d-f2fc9779-b3f6-4bec-8822-f5f6fe8041e0': <dask.highlevelgraph.MaterializedLayer at 0x16447d2a0>,
'finalize-8e4a37e5-68d7-4d57-908c-05611cb218a2': <dask.highlevelgraph.MaterializedLayer at 0x164d8dd20>,
'finalize-abb81368-05d0-48fd-bc81-4da3d383a026': <dask.highlevelgraph.MaterializedLayer at 0x164d8dd50>,
'getattr-d83101a6b9634af52ae2ed30c7d8dba0': <dask.highlevelgraph.MaterializedLayer at 0x16447d000>}
ml = client.get(merged_hgl,'add-a362066893b349d99b2b19a5d658fccb')
ml.__dir__()['annotations',
'collection_annotations',
'mapping',
'__module__',
'__doc__',
'__init__',
'__contains__',
'__getitem__',
'__iter__',
'__len__',
'is_materialized',
'get_output_keys',
'__abstractmethods__',
'_abc_impl',
'__slotnames__',
'__annotations__',
'cull',
'get_dependencies',
'clone',
'__copy__',
'_repr_html_',
'layer_info_dict',
'__orig_bases__',
'__dict__',
'__weakref__',
'__slots__',
'get',
'keys',
'items',
'values',
'__eq__',
'__reversed__',
'__hash__',
'__subclasshook__',
'__new__',
'__repr__',
'__str__',
'__getattribute__',
'__setattr__',
'__delattr__',
'__lt__',
'__le__',
'__ne__',
'__gt__',
'__ge__',
'__reduce_ex__',
'__reduce__',
'__init_subclass__',
'__format__',
'__sizeof__',
'__dir__',
'__class__',
'__class_getitem__']
dict(ml){'add-a362066893b349d99b2b19a5d658fccb': (<function _operator.add(a, b, /)>,
'add_d-a0be0541-c05e-46af-b8c5-df5fb17eea90',
'mul_d-f2fc9779-b3f6-4bec-8822-f5f6fe8041e0')}
merged_hglHighLevelGraph
HighLevelGraph with 1 layers and 12 keys from all layers.
<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">
<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>
</svg>
<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer1: 5977389568</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
5977389568
</p>
<table>
<tr>
<td>
<table>
<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">MaterializedLayer</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>
<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">12</td>
</tr>
</table>
</td>
<td>
</td>
</tr>
</table>
</details>
</div>
</div>
Dask Custom Graph
custom_graph = { "print" : (print("hello")) }
from dask.threaded import get
get(custom_graph, 'print')hello
graph = res.data.dask
dask_layers = graph.layers
extr = graph._all_external_keys
extr
layers = graph.layers
layers['add-ffe880f167c9131320013be9069dfca1'].__dir__()['annotations',
'collection_annotations',
'output',
'output_indices',
'output_blocks',
'dsk',
'numblocks',
'io_deps',
'indices',
'concatenate',
'new_axes',
'_dims',
'_cached_dict',
'__module__',
'__annotations__',
'__doc__',
'__init__',
'dims',
'__repr__',
'_dict',
'get_output_keys',
'__getitem__',
'__iter__',
'__len__',
'is_materialized',
'_cull_dependencies',
'_cull',
'cull',
'clone',
'__abstractmethods__',
'_abc_impl',
'get_dependencies',
'__copy__',
'_repr_html_',
'layer_info_dict',
'__orig_bases__',
'__dict__',
'__weakref__',
'__slots__',
'get',
'__contains__',
'keys',
'items',
'values',
'__eq__',
'__reversed__',
'__hash__',
'__subclasshook__',
'__new__',
'__str__',
'__getattribute__',
'__setattr__',
'__delattr__',
'__lt__',
'__le__',
'__ne__',
'__gt__',
'__ge__',
'__reduce_ex__',
'__reduce__',
'__init_subclass__',
'__format__',
'__sizeof__',
'__dir__',
'__class__',
'__class_getitem__']
dict(graph).keys()dict_keys([('add-ffe880f167c9131320013be9069dfca1', 0), ('add-ffe880f167c9131320013be9069dfca1', 1), ('add-ffe880f167c9131320013be9069dfca1', 2), ('add-ffe880f167c9131320013be9069dfca1', 3), ('add-d8368b0c413b7b1ff3e4545d5d219543', 0), ('add-d8368b0c413b7b1ff3e4545d5d219543', 1), ('add-d8368b0c413b7b1ff3e4545d5d219543', 2), ('add-d8368b0c413b7b1ff3e4545d5d219543', 3), ('array-93fac394d6028ca1841a0aaaf012fdfd', 0), ('array-93fac394d6028ca1841a0aaaf012fdfd', 1), ('array-93fac394d6028ca1841a0aaaf012fdfd', 2), ('array-93fac394d6028ca1841a0aaaf012fdfd', 3), ('pow-f98325ba6ded8b13c46607d966ba51d1', 0), ('pow-f98325ba6ded8b13c46607d966ba51d1', 1), ('pow-f98325ba6ded8b13c46607d966ba51d1', 2), ('pow-f98325ba6ded8b13c46607d966ba51d1', 3), ('array-e0289449abd23dc644f0fd074cd695dc', 0), ('array-e0289449abd23dc644f0fd074cd695dc', 1), ('array-e0289449abd23dc644f0fd074cd695dc', 2), ('array-e0289449abd23dc644f0fd074cd695dc', 3), ('pow-5022bc0a56d6e28f1fd172a750ba8cff', 0), ('pow-5022bc0a56d6e28f1fd172a750ba8cff', 1), ('pow-5022bc0a56d6e28f1fd172a750ba8cff', 2), ('pow-5022bc0a56d6e28f1fd172a750ba8cff', 3), ('mul-8e9e32e19f9c6e612b3800864d8cc810', 0), ('mul-8e9e32e19f9c6e612b3800864d8cc810', 1), ('mul-8e9e32e19f9c6e612b3800864d8cc810', 2), ('mul-8e9e32e19f9c6e612b3800864d8cc810', 3)])
for i in dict(graph):
print(get(dict(graph), i))
# 31, 52, 79, 112[31]
[52]
[79]
[112]
[26]
[40]
[58]
[80]
[1]
[2]
[3]
[4]
[1]
[4]
[9]
[16]
[5]
[6]
[7]
[8]
[25]
[36]
[49]
[64]
[5]
[12]
[21]
[32]
for i in extr:
print(get(extr, i[0]))3
0
2
2
1
2
1
3
0
3
2
2
2
2
1
2
0
2
1
3
3
0
2
2
3
2
3
3
xarray vs Numpy on delayed