{"id":9131,"date":"2022-07-12T16:10:56","date_gmt":"2022-07-12T21:10:56","guid":{"rendered":"https:\/\/www.rushworth.us\/lisa\/?p=9131"},"modified":"2022-07-12T16:10:57","modified_gmt":"2022-07-12T21:10:57","slug":"kibana-sankey-visualization","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=9131","title":{"rendered":"Kibana Sankey Visualization"},"content":{"rendered":"\n<p>Now that we&#8217;ve got a lot of data being ingested into our ELK platform, I am beginning to build out visualizations and dashboards. This Vega visualization (source below) shows the number of connections between source and destination countries. <\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><a href=\"https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/07\/kibana-vega-sankey.png\"><img loading=\"lazy\" decoding=\"async\" width=\"902\" height=\"398\" src=\"https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/07\/kibana-vega-sankey.png\" alt=\"\" class=\"wp-image-9132\" srcset=\"https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/07\/kibana-vega-sankey.png 902w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/07\/kibana-vega-sankey-300x132.png 300w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/07\/kibana-vega-sankey-768x339.png 768w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/07\/kibana-vega-sankey-750x331.png 750w\" sizes=\"auto, (max-width: 902px) 100vw, 902px\" \/><\/a><\/figure>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\n{ \n  $schema: https:\/\/vega.github.io\/schema\/vega\/v3.0.json\n  data: &#x5B;\n    {\n      \/\/ Respect currently selected time range and filter string\n      name: rawData\n      url: {\n        %context%: true\n        %timefield%: @timestamp\n        index: firewall_logs*\n        body: {\n          size: 0\n          aggs: {\n            table: {\n              composite: {\n                size: 10000\n                sources: &#x5B;\n                  {\n                    source_country: {\n                     terms: {field: &quot;srccountry.keyword&quot;}\n                    }\n                  }\n                  {\n                    dest_country: {\n                     terms: {field: &quot;dstcountry.keyword&quot;}\n                    }\n                  }\n                ]\n              }\n            }\n          }\n        }\n      }\n      format: {property: &quot;aggregations.table.buckets&quot;}\n      \/\/ Add aliases for data.* elements\n      transform: &#x5B;\n        {type: &quot;formula&quot;, expr: &quot;datum.key.source_country&quot;, as: &quot;source_country&quot;}\n        {type: &quot;formula&quot;, expr: &quot;datum.key.dest_country&quot;, as: &quot;dest_country&quot;}\n        {type: &quot;formula&quot;, expr: &quot;datum.doc_count&quot;, as: &quot;size&quot;}\n      ]\n    }\n    {\n      name: nodes\n      source: rawData\n      transform: &#x5B;\n        \/\/ Filter to selected country\n        {\n          type: filter\n          expr: !groupSelector || groupSelector.source_country == datum.source_country || groupSelector.dest_country == datum.dest_country\n        }\n        {type: &quot;formula&quot;, expr: &quot;datum.source_country+datum.dest_country&quot;, as: &quot;key&quot;}\n        {\n          type: fold\n          fields: &#x5B;&quot;source_country&quot;, &quot;dest_country&quot;]\n          as: &#x5B;&quot;stack&quot;, &quot;grpId&quot;]\n        }\n        {\n          type: formula\n          expr: datum.stack == &#039;source_country&#039; ? datum.source_country+&#039; &#039;+datum.dest_country : datum.dest_country+&#039; &#039;+datum.source_country\n          as: sortField\n        }\n        {\n          type: stack\n          groupby: &#x5B;&quot;stack&quot;]\n          sort: {field: &quot;sortField&quot;, order: &quot;descending&quot;}\n          field: size\n        }\n        {type: &quot;formula&quot;, expr: &quot;(datum.y0+datum.y1)\/2&quot;, as: &quot;yc&quot;}\n      ]\n    }\n    {\n      name: groups\n      source: nodes\n      transform: &#x5B;\n        \/\/ Aggregate country groups and include number of documents for each grouping\n        {\n          type: aggregate\n          groupby: &#x5B;&quot;stack&quot;, &quot;grpId&quot;]\n          fields: &#x5B;&quot;size&quot;]\n          ops: &#x5B;&quot;sum&quot;]\n          as: &#x5B;&quot;total&quot;]\n        }\n        {\n          type: stack\n          groupby: &#x5B;&quot;stack&quot;]\n          sort: {field: &quot;grpId&quot;, order: &quot;descending&quot;}\n          field: total\n        }\n        {type: &quot;formula&quot;, expr: &quot;scale(&#039;y&#039;, datum.y0)&quot;, as: &quot;scaledY0&quot;}\n        {type: &quot;formula&quot;, expr: &quot;scale(&#039;y&#039;, datum.y1)&quot;, as: &quot;scaledY1&quot;}\n        {type: &quot;formula&quot;, expr: &quot;datum.stack == &#039;source_country&#039;&quot;, as: &quot;rightLabel&quot;}\n        {\n          type: formula\n          expr: datum.total\/domain(&#039;y&#039;)&#x5B;1]\n          as: percentage\n        }\n      ]\n    }\n    {\n      name: destinationNodes\n      source: nodes\n      transform: &#x5B;\n        {type: &quot;filter&quot;, expr: &quot;datum.stack == &#039;dest_country&#039;&quot;}\n      ]\n    }\n    {\n      name: edges\n      source: nodes\n      transform: &#x5B;\n        {type: &quot;filter&quot;, expr: &quot;datum.stack == &#039;source_country&#039;&quot;}\n        {\n          type: lookup\n          from: destinationNodes\n          key: key\n          fields: &#x5B;&quot;key&quot;]\n          as: &#x5B;&quot;target&quot;]\n        }\n        {\n          type: linkpath\n          orient: horizontal\n          shape: diagonal\n          sourceY: {expr: &quot;scale(&#039;y&#039;, datum.yc)&quot;}\n          sourceX: {expr: &quot;scale(&#039;x&#039;, &#039;source_country&#039;) + bandwidth(&#039;x&#039;)&quot;}\n          targetY: {expr: &quot;scale(&#039;y&#039;, datum.target.yc)&quot;}\n          targetX: {expr: &quot;scale(&#039;x&#039;, &#039;dest_country&#039;)&quot;}\n        }\n        \/\/ Calculation to determine line thickness\n        {\n          type: formula\n          expr: range(&#039;y&#039;)&#x5B;0]-scale(&#039;y&#039;, datum.size)\n          as: strokeWidth\n        }\n        {\n          type: formula\n          expr: datum.size\/domain(&#039;y&#039;)&#x5B;1]\n          as: percentage\n        }\n      ]\n    }\n  ]\n  scales: &#x5B;\n    {\n      name: x\n      type: band\n      range: width\n      domain: &#x5B;&quot;source_country&quot;, &quot;dest_country&quot;]\n      paddingOuter: 0.05\n      paddingInner: 0.95\n    }\n    {\n      name: y\n      type: linear\n      range: height\n      domain: {data: &quot;nodes&quot;, field: &quot;y1&quot;}\n    }\n    {\n      name: color\n      type: ordinal\n      range: category\n      domain: {data: &quot;rawData&quot;, fields: &#x5B;&quot;source_country&quot;, &quot;dest_country&quot;]}\n    }\n    {\n      name: stackNames\n      type: ordinal\n      range: &#x5B;&quot;Source Country&quot;, &quot;Destination Country&quot;]\n      domain: &#x5B;&quot;source_country&quot;, &quot;dest_country&quot;]\n    }\n  ]\n  axes: &#x5B;\n    {\n      orient: bottom\n      scale: x\n      encode: {\n        labels: {\n          update: {\n            text: {scale: &quot;stackNames&quot;, field: &quot;value&quot;}\n          }\n        }\n      }\n    }\n    {orient: &quot;left&quot;, scale: &quot;y&quot;}\n  ]\n  marks: &#x5B;\n    {\n      type: path\n      name: edgeMark\n      from: {data: &quot;edges&quot;}\n      clip: true\n      encode: {\n        update: {\n          stroke: &#x5B;\n            {\n              test: groupSelector &amp;&amp; groupSelector.stack==&#039;source_country&#039;\n              scale: color\n              field: dest_country\n            }\n            {scale: &quot;color&quot;, field: &quot;source_country&quot;}\n          ]\n          strokeWidth: {field: &quot;strokeWidth&quot;}\n          path: {field: &quot;path&quot;}\n          strokeOpacity: {\n            signal: !groupSelector &amp;&amp; (groupHover.source_country == datum.source_country || groupHover.dest_country == datum.dest_country) ? 0.9 : 0.3\n          }\n          zindex: {\n            signal: !groupSelector &amp;&amp; (groupHover.source_country == datum.source_country || groupHover.dest_country == datum.dest_country) ? 1 : 0\n          }\n          tooltip: {\n            signal: datum.source_country + &#039; \u2192 &#039; + datum.dest_country + &#039;    &#039; + format(datum.size, &#039;,.0f&#039;) + &#039;   (&#039; + format(datum.percentage, &#039;.1%&#039;) + &#039;)&#039;\n          }\n        }\n        hover: {\n          strokeOpacity: {value: 1}\n        }\n      }\n    }\n    {\n      type: rect\n      name: groupMark\n      from: {data: &quot;groups&quot;}\n      encode: {\n        enter: {\n          fill: {scale: &quot;color&quot;, field: &quot;grpId&quot;}\n          width: {scale: &quot;x&quot;, band: 1}\n        }\n        update: {\n          x: {scale: &quot;x&quot;, field: &quot;stack&quot;}\n          y: {field: &quot;scaledY0&quot;}\n          y2: {field: &quot;scaledY1&quot;}\n          fillOpacity: {value: 0.6}\n          tooltip: {\n            signal: datum.grpId + &#039;   &#039; + format(datum.total, &#039;,.0f&#039;) + &#039;   (&#039; + format(datum.percentage, &#039;.1%&#039;) + &#039;)&#039;\n          }\n        }\n        hover: {\n          fillOpacity: {value: 1}\n        }\n      }\n    }\n    {\n      type: text\n      from: {data: &quot;groups&quot;}\n      interactive: false\n      encode: {\n        update: {\n          x: {\n            signal: scale(&#039;x&#039;, datum.stack) + (datum.rightLabel ? bandwidth(&#039;x&#039;) + 8 : -8)\n          }\n          yc: {signal: &quot;(datum.scaledY0 + datum.scaledY1)\/2&quot;}\n          align: {signal: &quot;datum.rightLabel ? &#039;left&#039; : &#039;right&#039;&quot;}\n          baseline: {value: &quot;middle&quot;}\n          fontWeight: {value: &quot;bold&quot;}\n          \/\/ Do not show labels on smaller items\n          text: {signal: &quot;abs(datum.scaledY0-datum.scaledY1) &gt; 13 ? datum.grpId : &#039;&#039;&quot;}\n        }\n      }\n    }\n    {\n      type: group\n      data: &#x5B;\n        {\n          name: dataForShowAll\n          values: &#x5B;{}]\n          transform: &#x5B;{type: &quot;filter&quot;, expr: &quot;groupSelector&quot;}]\n        }\n      ]\n      \/\/ Set button size and positioning\n      encode: {\n        enter: {\n          xc: {signal: &quot;width\/2&quot;}\n          y: {value: 30}\n          width: {value: 80}\n          height: {value: 30}\n        }\n      }\n      marks: &#x5B;\n        {\n          type: group\n          name: groupReset\n          from: {data: &quot;dataForShowAll&quot;}\n          encode: {\n            enter: {\n              cornerRadius: {value: 6}\n              fill: {value: &quot;#f5f5f5&quot;}\n              stroke: {value: &quot;#c1c1c1&quot;}\n              strokeWidth: {value: 2}\n              \/\/ use parent group&#039;s size\n              height: {\n                field: {group: &quot;height&quot;}\n              }\n              width: {\n                field: {group: &quot;width&quot;}\n              }\n            }\n            update: {\n              opacity: {value: 1}\n            }\n            hover: {\n              opacity: {value: 0.7}\n            }\n          }\n          marks: &#x5B;\n            {\n              type: text\n              interactive: false\n              encode: {\n                enter: {\n                  xc: {\n                    field: {group: &quot;width&quot;}\n                    mult: 0.5\n                  }\n                  yc: {\n                    field: {group: &quot;height&quot;}\n                    mult: 0.5\n                    offset: 2\n                  }\n                  align: {value: &quot;center&quot;}\n                  baseline: {value: &quot;middle&quot;}\n                  fontWeight: {value: &quot;bold&quot;}\n                  text: {value: &quot;Show All&quot;}\n                }\n              }\n            }\n          ]\n        }\n      ]\n    }\n  ]\n  signals: &#x5B;\n    {\n      name: groupHover\n      value: {}\n      on: &#x5B;\n        {\n          events: @groupMark:mouseover\n          update: &quot;{source_country:datum.stack==&#039;source_country&#039; &amp;&amp; datum.grpId, dest_country:datum.stack==&#039;dest_country&#039; &amp;&amp; datum.grpId}&quot;\n        }\n        {events: &quot;mouseout&quot;, update: &quot;{}&quot;}\n      ]\n    }\n    {\n      name: groupSelector\n      value: false\n      on: &#x5B;\n        {\n          events: @groupMark:click!\n          update: &quot;{stack:datum.stack, source_country:datum.stack==&#039;source_country&#039; &amp;&amp; datum.grpId, dest_country:datum.stack==&#039;dest_country&#039; &amp;&amp; datum.grpId}&quot;\n        }\n        {\n          events: &#x5B;\n            {type: &quot;click&quot;, markname: &quot;groupReset&quot;}\n            {type: &quot;dblclick&quot;}\n          ]\n          update: &quot;false&quot;\n        }\n      ]\n    }\n  ]\n}\n<\/pre><\/div>","protected":false},"excerpt":{"rendered":"<p>Now that we&#8217;ve got a lot of data being ingested into our ELK platform, I am beginning to build out visualizations and dashboards. This Vega visualization (source below) shows the number of connections between source and destination countries.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1588],"tags":[1591,1669,1668],"class_list":["post-9131","post","type-post","status-publish","format-standard","hentry","category-elk","tag-kibana","tag-vega","tag-visualizations"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9131","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9131"}],"version-history":[{"count":1,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9131\/revisions"}],"predecessor-version":[{"id":9133,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9131\/revisions\/9133"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9131"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9131"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9131"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}