index.xml

<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Just Numbers &amp; Things</title>
    <link>http://justnumbersandthings.com/</link>
      <atom:link href="http://justnumbersandthings.com/index.xml" rel="self" type="application/rss+xml" />
    <description>Just Numbers &amp; Things</description>
    <generator>Source Themes Academic (https://sourcethemes.com/academic/)</generator><language>en-us</language><copyright>© 2019</copyright><lastBuildDate>Sun, 24 Nov 2019 00:00:00 +0000</lastBuildDate>
    <image>
      <url>http://justnumbersandthings.com/img/icon-192.png</url>
      <title>Just Numbers &amp; Things</title>
      <link>http://justnumbersandthings.com/</link>
    </image>
    
    <item>
      <title> Setting up Oracle ODBC on macOS with ISQL</title>
      <link>http://justnumbersandthings.com/post/2019-11-24-setup-oracle-odbc-with-isql/</link>
      <pubDate>Sun, 24 Nov 2019 00:00:00 +0000</pubDate>
      <guid>http://justnumbersandthings.com/post/2019-11-24-setup-oracle-odbc-with-isql/</guid>
      <description>

&lt;h1 id=&#34;0-install-isql&#34;&gt;0) Install ISQL&lt;/h1&gt;

&lt;p&gt;If you&amp;rsquo;re on macOS the easiest way to install isql is using &lt;a href=&#34;https://brew.sh&#34; target=&#34;_blank&#34;&gt;homebrew&lt;/a&gt; &lt;code&gt;brew install unixodbc&lt;/code&gt;.&lt;/p&gt;

&lt;h1 id=&#34;1-download-drivers&#34;&gt;1) Download drivers&lt;/h1&gt;

&lt;ol&gt;
&lt;li&gt;Download ODBC Package &amp;amp; Basic Package from &lt;a href=&#34;https://www.oracle.com/database/technologies/instant-client/macos-intel-x86-downloads.html&#34; target=&#34;_blank&#34;&gt;here&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Create an oracle account if necessary&lt;/li&gt;
&lt;/ol&gt;

&lt;h1 id=&#34;2-prepare-driver-files&#34;&gt;2) Prepare driver files&lt;/h1&gt;

&lt;ol&gt;
&lt;li&gt;&lt;code&gt;cd Downloads&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;unzip instantclient-basic-macos.x64-19.3.0.0.0dbru.zip&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;unsip instantclient-odbc-macos.x64-19.3.0.0.0dbru.zip&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;mkdir ~/lib&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;mkdir -p /opt/oracle/&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;mv $(pwd)/instantclient_19_3 /opt/oracle/&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;ln -s /opt/oracle/instantclient_19_3/libclntsh.dylib.19.1 /opt/oracle/instantclient_19_3/libclntshcore.dylib.19.1 ~/lib&lt;/code&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h1 id=&#34;3-update-odbc-ini-files&#34;&gt;3) Update ODBC ini files&lt;/h1&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;update &lt;code&gt;odbcinst.ini&lt;/code&gt; to add&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;[ODBC Drivers]
Oracle ODBC Driver = Installed
...
...
[Oracle ODBC Driver]
Description = Oracle 19 ODBC driver
Driver      = /opt/oracle/instantclient_19_3/libsqora.dylib.19.1
Setup       =
FileUsage   =
CPTimeout   =
CPReuse     =
&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;

&lt;li&gt;&lt;p&gt;Update &lt;code&gt;odbc.ini&lt;/code&gt; to add&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;[ODBC Data Sources]
{dsn_name} = [Oracle ODBC Driver]
...
...
[{dsn_name}]
AggregateSQLType=FLOAT
Application Attributes=T
Attributes=W
BatchAutocommitMode=IfAllSuccessful
BindAsFLOAT=F
CacheBufferSize=20
CloseCursor=F
DisableDPM=F
DisableMTS=T
DisableRULEHint=T
Driver=/opt/oracle/instantclient_19_3/libsqora.dylib.19.1
DSN={dsn_name}
EXECSchemaOpt=
EXECSyntax=T
Failover=T
FailoverDelay=10
FailoverRetryCount=10
FetchBufferSize=64000
ForceWCHAR=F
LobPrefetchSize=8192
Lobs=T
Longs=T
MaxLargeData=0
MaxTokenSize=8192
MetadataIdDefault=F
QueryTimeout=T
ResultSets=T
ServerName={host}:{port}/{server}
SQLGetData extensions=F
SQLTranslateErrors=F
StatementCache=F
Translation DLL=
Translation Option=0
UseOCIDescribeAny=F
UserID={username}
Password={password}
&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h1 id=&#34;4-verify-installation&#34;&gt;4) Verify installation&lt;/h1&gt;

&lt;ol&gt;
&lt;li&gt;Verify by connecting &lt;code&gt;isql -v {dsn_name}&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Note: on macOS Catalina or later you may have to workaround the lack of notarization by navigating to &amp;ldquo;system preferences&amp;gt;security&amp;gt;allow&amp;rdquo; and approving multiple files after multiple connection attempts using step 4.1&lt;/li&gt;
&lt;/ol&gt;
</description>
    </item>
    
    <item>
      <title> Connecting to Google BigQuery with DBeaver</title>
      <link>http://justnumbersandthings.com/post/2018-10-07-dbeaver-bigquery-part2/</link>
      <pubDate>Sun, 07 Oct 2018 00:00:00 +0000</pubDate>
      <guid>http://justnumbersandthings.com/post/2018-10-07-dbeaver-bigquery-part2/</guid>
      <description>

&lt;h1 id=&#34;0-install-dbeaver&#34;&gt;0) Install DBeaver&lt;/h1&gt;

&lt;p&gt;You can find installation instructions &lt;a href=&#34;http://justnumbersandthings.com/post/2017-03-26-dbeaver-mac/&#34;&gt;here&lt;/a&gt;.
Make sure to install version 5.2.2 or later. If you haven&amp;rsquo;t updated to 5.2.2 or later you may use this &lt;a href=&#34;http://justnumbersandthings.com/post/2018-09-22-dbeaver-bigquery/&#34;&gt;post&lt;/a&gt;
as a guide for connecting to BigQuery.&lt;/p&gt;

&lt;h1 id=&#34;1-create-a-new-service-account&#34;&gt;1) Create a new service account&lt;/h1&gt;

&lt;ol&gt;
&lt;li&gt;Instructions &amp;amp; details for creating a new service account can be found on &lt;a href=&#34;https://cloud.google.com/docs/authentication/production#auth-cloud-implicit-python&#34; target=&#34;_blank&#34;&gt;Google&amp;rsquo;s website&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Grant your desired BigQuery permissions to your new service account&lt;/li&gt;
&lt;li&gt;Download the service account key&lt;/li&gt;
&lt;/ol&gt;

&lt;h1 id=&#34;2-create-a-new-connection&#34;&gt;2) Create a new connection&lt;/h1&gt;

&lt;ol&gt;
&lt;li&gt;In the menu bar navigate to &lt;code&gt;Database &amp;gt; New Connection&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Select BigQuery &amp;amp; press next&lt;/li&gt;
&lt;li&gt;Fill in project with the name of your BigQuery Project

&lt;ul&gt;
&lt;li&gt;Optional, add additional projects in the subsequent field&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;li&gt;Select service-based&lt;/li&gt;
&lt;li&gt;Fill in the name of the service account ex: &lt;code&gt;bigquery-demo@project_name.iam.gserviceaccount.com&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Fill in the path to the service key downloaded earlier&lt;/li&gt;
&lt;li&gt;Press finish&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Congrats you&amp;rsquo;ve successfully connected to BigQuery using Dbeaver!&lt;/p&gt;

&lt;h1 id=&#34;7-troubleshooting&#34;&gt;7) Troubleshooting&lt;/h1&gt;

&lt;ul&gt;
&lt;li&gt;If you receive
&lt;code&gt;[Simba][BigQueryJDBCDriver](100004) HttpTransport IO error : 403 Forbidden&lt;/code&gt; make sure you&amp;rsquo;ve created a service account with the necessary permissions.&lt;/li&gt;
&lt;/ul&gt;
</description>
    </item>
    
    <item>
      <title>Exploring Complex Types in BigQuery</title>
      <link>http://justnumbersandthings.com/post/2018-09-23-exploring-complex-types-bigquery/</link>
      <pubDate>Sun, 23 Sep 2018 00:00:00 +0000</pubDate>
      <guid>http://justnumbersandthings.com/post/2018-09-23-exploring-complex-types-bigquery/</guid>
      <description>

&lt;h1 id=&#34;introduction&#34;&gt;Introduction&lt;/h1&gt;

&lt;p&gt;Google&amp;rsquo;s BigQuery has support for complex types (arrays &amp;amp; structs) which are relatively new in analytical databases.
While the ideas and of arrays and structs aren&amp;rsquo;t unique to BigQuery some of the syntax and capabilities are unique.
In this post I&amp;rsquo;ll be going over what I&amp;rsquo;ve found to be the most useful patterns and tricks.&lt;/p&gt;

&lt;h2 id=&#34;arrays&#34;&gt;Arrays&lt;/h2&gt;

&lt;p&gt;Put plainly an array is a series of values of the same type stored within a single value.
You can create array literals via brackets &lt;code&gt;[]&lt;/code&gt; as demonstrated by the following snippet.&lt;/p&gt;

&lt;pre&gt;&lt;code class=&#34;language-sql&#34;&gt;select [1, 2, 3] as array_of_ints
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;You can also explicitly declare the type of an array as follows.&lt;/p&gt;

&lt;pre&gt;&lt;code class=&#34;language-sql&#34;&gt;select
  [&#39;2018-01-01&#39;, &#39;2018-02-01&#39;, &#39;2018-03-01&#39;] as array_of_string,
  array&amp;lt;date&amp;gt;[&#39;2018-01-01&#39;, &#39;2018-02-01&#39;, &#39;2018-03-01&#39;] as array_of_date
&lt;/code&gt;&lt;/pre&gt;

&lt;h2 id=&#34;structs&#34;&gt;Structs&lt;/h2&gt;

&lt;p&gt;A struct is a grouping of values that need not be of the same type and is very similar to the concept of tuples.
They are commonly used to group related values together.
You can create struct literals using the function struct&lt;/p&gt;

&lt;pre&gt;&lt;code class=&#34;language-sql&#34;&gt;select struct(1 as id, 2 as value) as user_info
&lt;/code&gt;&lt;/pre&gt;

&lt;h2 id=&#34;purpose&#34;&gt;Purpose&lt;/h2&gt;

&lt;p&gt;Arrays and structs allow for a more compact organization of related data which makes writing and reading many queries easier.
So while BigQuery is the engine that I&amp;rsquo;m covering today I expect these concepts to spread to other databases as knowledge of their utility spreads.&lt;/p&gt;

&lt;h1 id=&#34;basic-usage&#34;&gt;Basic Usage&lt;/h1&gt;

&lt;h2 id=&#34;example-dataset&#34;&gt;Example DataSet&lt;/h2&gt;

&lt;p&gt;I&amp;rsquo;ll be using the following CTE to demonstrate various functions for arrays &amp;amp; structs.&lt;/p&gt;

&lt;pre&gt;&lt;code class=&#34;language-sql&#34;&gt;with data_sample as (
  select
    1 as id_race,
    date&#39;2018-08-01&#39; as date_race,
    [3, 4] as id_participants,
    [ struct(7.0 as distance, 1 as lap_number, [struct(3 as id_participant, 1 as position),
                                                struct(4 as id_participant, 2 as position)] as finish_order),
      struct(6.5 as distance, 2 as lap_number, [struct(3 as id_participant, 1 as position),
                                                struct(4 as id_participant, 2 as position)] as finish_order),
      struct(7.2 as distance, 3 as lap_number, [struct(4 as id_participant, 1 as position),
                                                struct(3 as id_participant, 2 as position)] as finish_order)
      ] as race_laps
  union all
  select
    2 as id_race,
    date&#39;2018-08-08&#39; as date_race,
    [3, 5] as id_participants,
    [ struct(7.5 as distance, 1 as lap_number, [struct(5 as id_participant, 1 as position),
                                                struct(3 as id_participant, 2 as position)] as finish_order),
      struct(7.4 as distance, 2 as lap_number, [struct(5 as id_participant, 1 as position),
                                                struct(3 as id_participant, 2 as position)] as finish_order),
      struct(7.3 as distance, 3 as lap_number, [struct(5 as id_participant, 1 as position),
                                                struct(3 as id_participant, 2 as position)] as finish_order)
      ] as race_laps
)
&lt;/code&gt;&lt;/pre&gt;

&lt;h2 id=&#34;access-an-individual-element-from-an-array&#34;&gt;Access an individual element from an array&lt;/h2&gt;

&lt;pre&gt;&lt;code class=&#34;language-sql&#34;&gt;with data_sample as (
  --See above
  --Note: these lines will be omitted from subsequent examples
)
select
  ds.id_participants[offset(0)] as first_participant, --zero based
  ds.id_participants[ordinal(1)] as first_participant_also --one based
from data_sample as ds
&lt;/code&gt;&lt;/pre&gt;

&lt;h2 id=&#34;determine-the-length-of-an-array&#34;&gt;Determine the length of an array&lt;/h2&gt;

&lt;pre&gt;&lt;code class=&#34;language-sql&#34;&gt;select
  array_length(ds.id_participants) as number_of_participants
from data_sample as ds
&lt;/code&gt;&lt;/pre&gt;

&lt;h2 id=&#34;access-values-structs-within-an-array&#34;&gt;Access values &amp;amp; structs within an array&lt;/h2&gt;

&lt;p&gt;There are a couple of different ways to interact with arrays in BigQuery. The following three examples show different ways to access the example data structure and calculate the total distance for each race.&lt;/p&gt;

&lt;h3 id=&#34;by-joining-the-lap-array&#34;&gt;by joining the lap array&lt;/h3&gt;

&lt;pre&gt;&lt;code class=&#34;language-sql&#34;&gt;select
  ds.id_race,
  sum(rl.distance) as race_distance
from data_sample as ds
join ds.race_laps as rl
group by 1
&lt;/code&gt;&lt;/pre&gt;

&lt;h3 id=&#34;by-unnesting-the-lap-array&#34;&gt;by unnesting the lap array&lt;/h3&gt;

&lt;pre&gt;&lt;code class=&#34;language-sql&#34;&gt;select
  ds.id_race,
  sum(rl.distance) as race_distance
from data_sample as ds,
unnest(ds.race_laps) as rl
group by 1
&lt;/code&gt;&lt;/pre&gt;

&lt;h3 id=&#34;by-using-at-inline-query&#34;&gt;by using at inline query&lt;/h3&gt;

&lt;pre&gt;&lt;code class=&#34;language-sql&#34;&gt;select
  ds.id_race,
  (select sum(rl.distance) from unnest(ds.race_laps) as rl) as race_distance
from data_sample as ds
&lt;/code&gt;&lt;/pre&gt;

&lt;h3 id=&#34;by-joining-multiple-arrays&#34;&gt;by joining multiple arrays&lt;/h3&gt;

&lt;p&gt;The following query returns a list of race ids &amp;amp; participants ids &amp;amp; and a comma separated string showing that participants place each lap of each race.&lt;/p&gt;

&lt;pre&gt;&lt;code class=&#34;language-sql&#34;&gt;select
  ds.id_race,
  fo.id_participant,
  string_agg(cast(fo.position as string), &#39;, &#39; order by rl.lap_number) lap_positions
from data_sample as ds
join ds.race_laps as rl
join rl.finish_order as fo
group by 1, 2
&lt;/code&gt;&lt;/pre&gt;

&lt;h2 id=&#34;filtering-by-contents-of-an-array&#34;&gt;Filtering by contents of an array&lt;/h2&gt;

&lt;pre&gt;&lt;code class=&#34;language-sql&#34;&gt;select
  ds.id_race
from data_sample as ds
where 3 in unnest(ds.id_participants)
&lt;/code&gt;&lt;/pre&gt;
</description>
    </item>
    
    <item>
      <title> Connecting to Google BigQuery with DBeaver with JDBC Drivers</title>
      <link>http://justnumbersandthings.com/post/2018-09-22-dbeaver-bigquery/</link>
      <pubDate>Sat, 22 Sep 2018 00:00:00 +0000</pubDate>
      <guid>http://justnumbersandthings.com/post/2018-09-22-dbeaver-bigquery/</guid>
      <description>

&lt;h1 id=&#34;update-2018-10-07&#34;&gt;Update (2018-10-07)&lt;/h1&gt;

&lt;p&gt;Shortly after this post DBeaver was updated with a native connector.
Please see this &lt;a href=&#34;http://justnumbersandthings.com/post/2018-10-07-dbeaver-bigquery-part2/&#34;&gt;post&lt;/a&gt; for a more up to date connection instructions if
you have updated DBeaver to 5.2.2 or later.&lt;/p&gt;

&lt;h1 id=&#34;0-install-dbeaver&#34;&gt;0) Install DBeaver&lt;/h1&gt;

&lt;p&gt;You can find installation instructions &lt;a href=&#34;http://justnumbersandthings.com/post/2017-03-26-dbeaver-mac/&#34;&gt;here&lt;/a&gt;&lt;/p&gt;

&lt;h1 id=&#34;1-download-the-latest-drivers&#34;&gt;1) Download the latest drivers&lt;/h1&gt;

&lt;p&gt;You can find the latest drivers on &lt;a href=&#34;https://cloud.google.com/bigquery/partners/simba-drivers/&#34; target=&#34;_blank&#34;&gt;Google&amp;rsquo;s website&lt;/a&gt;&lt;/p&gt;

&lt;h1 id=&#34;2-create-a-folder-to-store-the-drivers&#34;&gt;2) Create a folder to store the drivers&lt;/h1&gt;

&lt;p&gt;&lt;code&gt;mkdir ~/.dbeaver-drivers/bigquery/&lt;/code&gt;&lt;/p&gt;

&lt;h1 id=&#34;3-extract-driver-jars-and-move-to-the-folder-we-made-earlier&#34;&gt;3) Extract driver jars and move to the folder we made earlier&lt;/h1&gt;

&lt;p&gt;&lt;img src=&#34;http://justnumbersandthings.com/img/20180922_bigquery_jars_in_place.png&#34; alt=&#34;jars_in_folder&#34; /&gt;&lt;/p&gt;

&lt;h1 id=&#34;4-create-a-new-driver-in-dbeaver&#34;&gt;4) Create a New Driver in DBeaver&lt;/h1&gt;

&lt;ol&gt;
&lt;li&gt;Navigate to &lt;code&gt;Database &amp;gt; Driver Manager &amp;gt; New&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Add all the files from &lt;code&gt;~/.dbeaver-drivers/bigquery/&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Driver name: &lt;code&gt;BigQuery&lt;/code&gt; (for labeling only)&lt;/li&gt;
&lt;li&gt;Class name: &lt;code&gt;com.simba.googlebigquery.jdbc42.Driver&lt;/code&gt; (at the time of this writing)&lt;/li&gt;
&lt;li&gt;Default port: &lt;code&gt;443&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;URL template: &lt;code&gt;jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId={server};OAuthType=0;OAuthServiceAcctEmail={user};OAuthPvtKeyPath={host};&lt;/code&gt;

&lt;ul&gt;
&lt;li&gt;Note there are 4 different ways to connect to BigQuery using the JDBC driver. This tutorial illustrates connecting using the service account authorization method.&lt;/li&gt;
&lt;li&gt;Additionally, at the time of this writing, Dbeaver only supports a couple URL template variables (e.g. server, user, host) which is why I&amp;rsquo;ve used the &lt;code&gt;host&lt;/code&gt; variable for a path to key file rather than something like &lt;code&gt;key_path&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h1 id=&#34;5-create-a-new-service-account&#34;&gt;5) Create a new service account&lt;/h1&gt;

&lt;ol&gt;
&lt;li&gt;Instructions &amp;amp; details for creating a new service account can be found on &lt;a href=&#34;https://cloud.google.com/docs/authentication/production#auth-cloud-implicit-python&#34; target=&#34;_blank&#34;&gt;Google&amp;rsquo;s website&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Grant your desired BigQuery permissions to your new service account&lt;/li&gt;
&lt;li&gt;Download the service account key&lt;/li&gt;
&lt;/ol&gt;

&lt;h1 id=&#34;6-create-a-new-connection&#34;&gt;6) Create a New Connection&lt;/h1&gt;

&lt;ol&gt;
&lt;li&gt;In the menu bar navigate to &lt;code&gt;Database &amp;gt; New Connection&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Select BigQuery&lt;/li&gt;
&lt;li&gt;Fill in the appropriate values for host, server, user&lt;/li&gt;
&lt;li&gt;Set host to the path the service account key e.g., &lt;code&gt;/Users/admin/.dbeaver_drivers/bigquery/project_name-####.json&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Set server to project id for your BigQuery project&lt;/li&gt;
&lt;li&gt;Set user to the email address for the generated service account&lt;/li&gt;
&lt;li&gt;Press finish&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Congrats you&amp;rsquo;ve successfully connected to BigQuery using Dbeaver!&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;http://justnumbersandthings.com/img/20180922_bigquery_connection.png&#34; alt=&#34;successful connection&#34; /&gt;&lt;/p&gt;

&lt;h1 id=&#34;7-troubleshooting&#34;&gt;7) Troubleshooting&lt;/h1&gt;

&lt;ul&gt;
&lt;li&gt;If you receive &lt;code&gt;[Simba][BigQueryJDBCDriver](100004) HttpTransport IO error : 403 Forbidden&lt;/code&gt; make sure you&amp;rsquo;ve created a service account with the necessary permissions.&lt;/li&gt;
&lt;li&gt;-As of Dbeaver 5.2.0 Dbeaver is unable to return arrays of ints and other numerics-

&lt;ul&gt;
&lt;li&gt;e.g., select [1, 2] as ids results in &lt;code&gt;org.jkiss.dbeaver.model.exec.DBCException: SQL Error [10140] [22003]: [Simba][JDBC](10140) Error converting value to long.&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;/ul&gt;
</description>
    </item>
    
    <item>
      <title>Importing a CSV into a database using DBeaver Part 2</title>
      <link>http://justnumbersandthings.com/post/2018-08-12-dbeaver-import-csv-2/</link>
      <pubDate>Sun, 12 Aug 2018 00:00:00 +0000</pubDate>
      <guid>http://justnumbersandthings.com/post/2018-08-12-dbeaver-import-csv-2/</guid>
      <description>

&lt;h2 id=&#34;forward&#34;&gt;Forward&lt;/h2&gt;

&lt;p&gt;This is a followup to my previous &lt;a href=&#34;http://justnumbersandthings.com/post/2018-06-12-dbeaver-import-csv/&#34;&gt;post&lt;/a&gt;. My previous post demonstrated how to import a CSV using Dbeaver&amp;rsquo;s database to database export &amp;amp; import feature. As of version 5.1.5 Dbeaver introduced a direct CSV option for importing CSVs.&lt;/p&gt;

&lt;h2 id=&#34;0-install-dbeaver&#34;&gt;0) Install DBeaver&lt;/h2&gt;

&lt;p&gt;You can find installation instructions &lt;a href=&#34;http://justnumbersandthings.com/post/2017-03-26-dbeaver-mac/&#34;&gt;here&lt;/a&gt;&lt;/p&gt;

&lt;h2 id=&#34;1-connect-to-your-target-database&#34;&gt;1) Connect to your target database&lt;/h2&gt;

&lt;p&gt;1.1) Navigate through your target database &amp;amp; schema and right click on your target table and select import table data&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;http://justnumbersandthings.com/img/20180612_import_table_data.png&#34; alt=&#34;import_table_data&#34; /&gt;&lt;/p&gt;

&lt;p&gt;1.2) Next select CSV from the list&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;http://justnumbersandthings.com/img/20180812_select_csv_option.png&#34; alt=&#34;select_csv_option&#34; /&gt;&lt;/p&gt;

&lt;p&gt;1.3) Select your CSV file for upload&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;http://justnumbersandthings.com/img/20180812_select_csv_file.png&#34; alt=&#34;select_csv_table&#34; /&gt;&lt;/p&gt;

&lt;h2 id=&#34;2-ensure-that-the-mappings-of-each-of-your-columns-is-correct&#34;&gt;2) Ensure that the mappings of each of your columns is correct&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;For column names that are an exact match DBeaver will automatically map them for you&lt;/li&gt;
&lt;li&gt;For the remaining columns make sure to map the source columns to your desired target columns&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;img src=&#34;http://justnumbersandthings.com/img/20180812_map_columns.png&#34; alt=&#34;map_columns&#34; /&gt;&lt;/p&gt;

&lt;h2 id=&#34;3-complete-the-wizard-and-watch-dbeaver-import-your-data&#34;&gt;3) Complete the wizard and watch DBeaver import your data&lt;/h2&gt;

&lt;p&gt;Note: For large files it may be necessary to go get lunch but in my case 4 records doesn&amp;rsquo;t take long to import :)&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;http://justnumbersandthings.com/img/20180612_data_finished.png&#34; alt=&#34;data_finished&#34; /&gt;&lt;/p&gt;

&lt;h2 id=&#34;4-check-to-make-sure-that-the-data-has-loaded-correctly&#34;&gt;4) Check to make sure that the data has loaded correctly&lt;/h2&gt;

&lt;p&gt;As a last optional step it is good practice to make sure that everything loaded correctly which can easily be done by running a query against your target DB&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;http://justnumbersandthings.com/img/20180612_data_in_table.png&#34; alt=&#34;data_in_table&#34; /&gt;&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Importing a CSV into a database using DBeaver</title>
      <link>http://justnumbersandthings.com/post/2018-06-12-dbeaver-import-csv/</link>
      <pubDate>Tue, 12 Jun 2018 00:00:00 +0000</pubDate>
      <guid>http://justnumbersandthings.com/post/2018-06-12-dbeaver-import-csv/</guid>
      <description>

&lt;h2 id=&#34;update-august-12-2018&#34;&gt;Update: August 12, 2018&lt;/h2&gt;

&lt;p&gt;The following post demonstrates how to import CSVs using Dbeaver&amp;rsquo;s database to database export &amp;amp; import feature. If you are certain of the quality of your CSV &amp;amp; just want to import it quickly my subsequent &lt;a href=&#34;http://justnumbersandthings.com/post/2018-08-12-dbeaver-import-csv-2/&#34;&gt;post&lt;/a&gt; may be more useful.&lt;/p&gt;

&lt;h2 id=&#34;0-install-dbeaver&#34;&gt;0) Install DBeaver&lt;/h2&gt;

&lt;p&gt;You can find installation instructions &lt;a href=&#34;http://justnumbersandthings.com/post/2017-03-26-dbeaver-mac/&#34;&gt;here&lt;/a&gt;&lt;/p&gt;

&lt;h2 id=&#34;1-create-a-folder-to-be-used-as-your-csv-database&#34;&gt;1) Create a folder to be used as your CSV Database&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;mkdir ~/desktop/csvs&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;Place the CSV you want to load into this folder&lt;/p&gt;

&lt;h2 id=&#34;2-create-a-csv-database-connection&#34;&gt;2) Create a CSV database connection&lt;/h2&gt;

&lt;p&gt;In the menu bar select &lt;code&gt;Database &amp;gt; Create a New Connection&lt;/code&gt;
 &amp;amp; from the list of drivers select &lt;code&gt;Flat files(CSV) &amp;gt; CSV/DBF&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;http://justnumbersandthings.com/img/20180612_select_driver.png&#34; alt=&#34;select_driver&#34; /&gt;&lt;/p&gt;

&lt;p&gt;Set the path of the connection to the folder you created earlier (the JDBC URL will auto-populate)&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;http://justnumbersandthings.com/img/20180612_configure_path.png&#34; alt=&#34;configure_path&#34; /&gt;&lt;/p&gt;

&lt;p&gt;Note: If you run into trouble downloading the driver navigate to the &lt;a href=&#34;http://csvjdbc.sourceforge.net/&#34; target=&#34;_blank&#34;&gt;source website&lt;/a&gt; and download the driver manually&lt;/p&gt;

&lt;h2 id=&#34;3-connect-to-your-target-database&#34;&gt;3) Connect to your target database&lt;/h2&gt;

&lt;p&gt;3.1) Navigate through your target database &amp;amp; schema and right click on your target table and select import table data&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;http://justnumbersandthings.com/img/20180612_import_table_data.png&#34; alt=&#34;import_table_data&#34; /&gt;&lt;/p&gt;

&lt;p&gt;3.2) Next select your source CSV from your CSV connection as the source container&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;http://justnumbersandthings.com/img/20180612_select_csv_table.png&#34; alt=&#34;select_csv_table&#34; /&gt;&lt;/p&gt;

&lt;p&gt;Note: In this example case I&amp;rsquo;m loading a test CSV into a Postgres database but this functionality works with any connection that DBeaver supports (which is basically everything)&lt;/p&gt;

&lt;h2 id=&#34;4-ensure-that-the-mappings-of-each-of-your-columns-is-correct&#34;&gt;4) Ensure that the mappings of each of your columns is correct&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;For column names that are an exact match DBeaver will automatically map them for you&lt;/li&gt;
&lt;li&gt;For the remaining columns make sure to map the source columns to your desired target columns&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;img src=&#34;http://justnumbersandthings.com/img/20180612_map_columns.png&#34; alt=&#34;map_columns&#34; /&gt;&lt;/p&gt;

&lt;h2 id=&#34;5-complete-the-wizard-and-watch-dbeaver-import-your-data&#34;&gt;5) Complete the wizard and watch DBeaver import your data&lt;/h2&gt;

&lt;p&gt;Note: For large files it may be necessary to go get lunch but in my case 4 records doesn&amp;rsquo;t take long to import :)&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;http://justnumbersandthings.com/img/20180612_data_finished.png&#34; alt=&#34;data_finished&#34; /&gt;&lt;/p&gt;

&lt;h2 id=&#34;6-check-to-make-sure-that-the-data-has-loaded-correctly&#34;&gt;6) Check to make sure that the data has loaded correctly&lt;/h2&gt;

&lt;p&gt;As a last optional step it is good practice to make sure that everything loaded correctly which can easily be done by running a query against your target DB&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;http://justnumbersandthings.com/img/20180612_data_in_table.png&#34; alt=&#34;data_in_table&#34; /&gt;&lt;/p&gt;

&lt;h2 id=&#34;7-final-notes-thoughts&#34;&gt;7) Final Notes &amp;amp; Thoughts&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;While this process takes a little bit more time to get setup than other tools setting up the CSV connection only needs to be done once

&lt;ul&gt;
&lt;li&gt;One side benefit of this as well is that you are now able to run SQL queries against CSVs very easily&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;li&gt;The only real pain point that I have run across is that if you add a new CSV file or add/delete columns in an active CSV connection you have to cancel the import wizard &amp;amp; refresh the CSV connection for the changes to be picked up

&lt;ul&gt;
&lt;li&gt;this feedback was provided in &lt;a href=&#34;https://github.com/dbeaver/dbeaver/issues/926&#34; target=&#34;_blank&#34;&gt;issue 926&lt;/a&gt; and hopefully it will be resolved in a future update&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;/ul&gt;
</description>
    </item>
    
    <item>
      <title>Moved to Hugo</title>
      <link>http://justnumbersandthings.com/post/2018-03-23-hugo-migration/</link>
      <pubDate>Fri, 23 Mar 2018 00:00:00 +0000</pubDate>
      <guid>http://justnumbersandthings.com/post/2018-03-23-hugo-migration/</guid>
      <description>

&lt;h2 id=&#34;what-was-i-using-before&#34;&gt;What was I using before&lt;/h2&gt;

&lt;p&gt;Previously I was using &lt;a href=&#34;https://github.com/getpelican/pelican&#34; target=&#34;_blank&#34;&gt;Pelican&lt;/a&gt; a static site generator written in Python. Personally I&amp;rsquo;m a huge Python fan which led me to search out Pelican rather than go with a more popular solution such as &lt;a href=&#34;https://jekyllrb.com/&#34; target=&#34;_blank&#34;&gt;Jekyll&lt;/a&gt;.&lt;/p&gt;

&lt;h2 id=&#34;why-change&#34;&gt;Why Change&lt;/h2&gt;

&lt;p&gt;As it turns out if you aren&amp;rsquo;t going to be modifying the static site generator in anyway you don&amp;rsquo;t need to concern yourself with the language that it is written in. This fact escaped me while I was seeking out Pelican. Coming to this realization opened up a world of possibilities and when I saw an article discussing Hugo and its killer feature, &lt;a href=&#34;https://gohugo.io/getting-started/usage/#livereload&#34; target=&#34;_blank&#34;&gt;Live Reload&lt;/a&gt;, I made the jump!&lt;/p&gt;

&lt;h2 id=&#34;unique-migration-tasks&#34;&gt;Unique Migration tasks&lt;/h2&gt;

&lt;p&gt;The most complex task of this migration for me was ensuring that the old urls would point to the correct articles after I moved. fortunately Hugo made this dead simple via url &lt;a href=&#34;https://gohugo.io/content-management/urls/#aliases&#34; target=&#34;_blank&#34;&gt;Aliases&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Simply add the following to the front matter of the article you&amp;rsquo;d like migrate.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;+++
aliases = [
    &amp;quot;/posts/my-original-url/&amp;quot;,
    &amp;quot;/2010/01/01/even-earlier-url.html&amp;quot;
]
+++
&lt;/code&gt;&lt;/pre&gt;
</description>
    </item>
    
    <item>
      <title>macOS 10.13.2 EGPU Adventures</title>
      <link>http://justnumbersandthings.com/post/2017-12-28-egpu-adventures/</link>
      <pubDate>Thu, 28 Dec 2017 00:00:00 +0000</pubDate>
      <guid>http://justnumbersandthings.com/post/2017-12-28-egpu-adventures/</guid>
      <description>

&lt;h1 id=&#34;1-what-is-an-egpu-when-did-apple-start-supporting-them&#34;&gt;1) What is an EGPU &amp;amp; when did Apple start supporting them?&lt;/h1&gt;

&lt;p&gt;The acronym EGPU stands for external &lt;a href=&#34;https://en.wikipedia.org/wiki/Graphics_processing_unit&#34; target=&#34;_blank&#34;&gt;GPU&lt;/a&gt;. Until recently they were relatively niche however, with the release of thunderbolt3 they have begun to gain traction. At WWDC17 Apple announced that they would be fully supported in High Sierra sometime during the spring of 2018. This news was welcomed at &lt;a href=&#34;egpu.io&#34; target=&#34;_blank&#34;&gt;egpu.io&lt;/a&gt; where a community of capable individuals have been trail blazing EGPUs on macOS &lt;a href=&#34;https://EGPU.io/setup-guide-external-graphics-card-mac/&#34; target=&#34;_blank&#34;&gt;prior to 10.13&lt;/a&gt; &amp;amp; Windows. They quickly updated some of their &lt;a href=&#34;https://EGPU.io/macos-high-sierra-official-external-gpu/&#34; target=&#34;_blank&#34;&gt;guides and recommendations&lt;/a&gt; to help beginners, such as myself, start their EGPU journey on High Sierra.&lt;/p&gt;

&lt;h1 id=&#34;2-my-setup&#34;&gt;2) My setup&lt;/h1&gt;

&lt;p&gt;So after reading about the success and watching several youtube reviews I thought it was time to give this EGPU thing a go. For my enclosure I decided to go with the &lt;a href=&#34;http://amzn.to/2CgKwQR&#34; target=&#34;_blank&#34;&gt;Mantiz Venus&lt;/a&gt; ($399). For my graphics card I decided to go with the &lt;a href=&#34;http://amzn.to/2Ci1wpR&#34; target=&#34;_blank&#34;&gt;Vega64&lt;/a&gt;. As a side note it looks like some users have had success with Nvidia cards on macOS after some elbow grease but AMD cards were reported to be plug and play; hence my decisions to go with AMD.&lt;/p&gt;

&lt;h1 id=&#34;3-initial-experiences&#34;&gt;3) Initial experiences&lt;/h1&gt;

&lt;p&gt;After plugging it in and turning it on for the first time I was excited to see that everything was working out of the box.&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;http://justnumbersandthings.com/img/20171228_egpu_plug_n_play.png&#34; alt=&#34;plug_n_play&#34; /&gt;&lt;/p&gt;

&lt;p&gt;The only oddity that I noticed at first is that system information lists the Vega64 as &amp;ldquo;AMD RX xxx&amp;rdquo;. However, his appears to only be a cosmetic issue so I ignore it.&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;http://justnumbersandthings.com/img/20171228_system_information.png&#34; alt=&#34;system_information&#34; /&gt;&lt;/p&gt;

&lt;p&gt;So after my initial excitement I tried playing a couple games with my new setup and quickly stumbled across something I should have uncovered sooner in my research. The Vega64 is quite power hungry and the enclosure I purchased &lt;a href=&#34;https://myMantiz.com/pages/faq&#34; target=&#34;_blank&#34;&gt;doesn&amp;rsquo;t actually support it, it only supports Vega56&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;For reference the problem that you&amp;rsquo;ll encounter is that the entire enclosure will shut down which results in a soft or sometimes a hard system crash. So for those are thinking about purchasing the Mantiz &amp;amp; Vega make sure to go with the Vega56 rather than the Vega64. However, if you are like me and already have a Vega64 and/or you want to try an unsupported setup keep reading.&lt;/p&gt;

&lt;h1 id=&#34;4-resolving-the-power-draw-issue&#34;&gt;4) Resolving the power draw issue.&lt;/h1&gt;

&lt;p&gt;After some more research I found a new &lt;a href=&#34;https://www.newegg.com/Product/Product.aspx?Item=N82E16817273013&#34; target=&#34;_blank&#34;&gt;power supply&lt;/a&gt; ($190) that would provide the needed power &amp;amp; had the same dimensions as the power supply that ships with the Mantiz. I&amp;rsquo;m happy to report that installing this part turned out to be essentially painless. The existing power supply comes out without much fuss and the new power supply fits perfectly in place of the old one. The only exception here is that the new power supply is a little longer than the existing one but there is ample room in the enclosure to accommodate this.&lt;/p&gt;

&lt;h1 id=&#34;5-subsequent-experiences&#34;&gt;5) Subsequent experiences&lt;/h1&gt;

&lt;p&gt;After upgrading the power supply I&amp;rsquo;ve yet to experience another power related crash and have had a far more enjoyable experience.&lt;/p&gt;

&lt;p&gt;The only remaining issue that I&amp;rsquo;ve had is a rare crash. Twice I&amp;rsquo;ve had macOS freeze for several seconds followed by a flash which results in all of my applications being closed (according to the dock, force-quite dialogs, &amp;amp; command-tab) but leaving the underlying processes untouched. This results in a bizarre situation where I can&amp;rsquo;t view my application however, I can still hear the audio and see the process running in activity monitor. I expect/hope that this is one of those things that will be squashed once Apple rolls out official EGPU support in the spring with 10.13.x.&lt;/p&gt;

&lt;h1 id=&#34;6-benchmarks&#34;&gt;6) Benchmarks&lt;/h1&gt;

&lt;p&gt;What review of an EGPU setup would be complete without some benchmarks? Given the wide variety of applications &amp;amp; uses for an EGPU I decided to stay simple with some Geekbench benchmarks.&lt;/p&gt;

&lt;p&gt;Using the AMD Pro 555 in my macbook-pro I received a score of &lt;a href=&#34;https://browser.geekbench.com/v4/compute/1670314&#34; target=&#34;_blank&#34;&gt;37,697&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;http://justnumbersandthings.com/img/20171228_internal_score.png&#34; alt=&#34;internal_score&#34; /&gt;&lt;/p&gt;

&lt;p&gt;Using my EGPU setup I received a score of &lt;a href=&#34;https://browser.geekbench.com/v4/compute/1670320&#34; target=&#34;_blank&#34;&gt;175,303&lt;/a&gt; a 4.65x improvement over the internal GPU offered by Apple. Quite the improvement!&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;http://justnumbersandthings.com/img/20171228_external_score.png&#34; alt=&#34;external_score&#34; /&gt;&lt;/p&gt;

&lt;h1 id=&#34;7-final-thoughts&#34;&gt;7) Final Thoughts&lt;/h1&gt;

&lt;h2 id=&#34;pros&#34;&gt;Pros&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Performance

&lt;ul&gt;
&lt;li&gt;The Geekbench results show a 4.65x increase and my experiences with this increase have been tremendously positive&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;li&gt;Flexibility

&lt;ul&gt;
&lt;li&gt;If I need my laptop to be portable it is as simple as unplugging a cord&lt;/li&gt;
&lt;li&gt;If I need my laptop to be ultra-fast it is as simple as plugging in a cord&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;li&gt;Upgradability

&lt;ul&gt;
&lt;li&gt;With this setup my Apple laptop now has an upgradable GPU which means I can keep my machine going for longer&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;li&gt;Ports

&lt;ul&gt;
&lt;li&gt;One of the nice things about the Mantiz enclosure is that it provides a lot of ports (5 USB &amp;amp; 1 gigabit ethernet port) and even includes a spot for a &lt;a href=&#34;https://www.amazon.com/gp/search?ie=UTF8&amp;amp;tag=justnumbersan-20&amp;amp;linkCode=ur2&amp;amp;linkId=f73bf29f7e5ed526f77b4bfb06ae1826&amp;amp;camp=1789&amp;amp;creative=9325&amp;amp;index=electronics&amp;amp;keywords=SSD&#34; target=&#34;_blank&#34;&gt;2.5&amp;rdquo; SSD Drive&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;So rather than buying more dongles you can utilize the Mantiz as a hub as well&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&#34;cons&#34;&gt;Cons&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Weight

&lt;ul&gt;
&lt;li&gt;One thing that caught me off guard was just how heavy the enclosure &amp;amp; GPU were&lt;/li&gt;
&lt;li&gt;It is far lighter than a desktop tower but still heavy enough to dissuade any thoughts of lugging it around&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;li&gt;Drivers

&lt;ul&gt;
&lt;li&gt;I thought that the Vega series would have first class support as an EGPU because of the Vega&amp;rsquo;s inclusion in the iMac Pro however, since system information doesn&amp;rsquo;t accurately identify the card it is clear that some support is still yet to come, likely in the spring&lt;/li&gt;
&lt;li&gt;Update as of 10.13.4 This has been resolved with Apple now officially supporting EGPU setups&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;li&gt;Support (Apple now officially supports EGPU setups for AMD cards)

&lt;ul&gt;
&lt;li&gt;&lt;del&gt;If you go with the Vega or really any other setup other than the &lt;a href=&#34;https://developer.apple.com/development-kit/external-graphics/&#34; target=&#34;_blank&#34;&gt;Apple EGPU kit&lt;/a&gt; you are in an area that isn&amp;rsquo;t officially supported&lt;/del&gt;&lt;/li&gt;
&lt;li&gt;&lt;del&gt;That being said you won&amp;rsquo;t be alone or the first so don&amp;rsquo;t let that hold you back too much&lt;/del&gt;&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In summary I&amp;rsquo;d recommend an EGPU to anyone who is looking for a more robust GPU. The Pros are very compelling and the cons while worth considering don&amp;rsquo;t out weight the benefits for anyone looking for improved performance. The only other thing I&amp;rsquo;ll call out is that I&amp;rsquo;d recommend the &lt;a href=&#34;http://amzn.to/2C4u4Gw&#34; target=&#34;_blank&#34;&gt;Vega56&lt;/a&gt; + &lt;a href=&#34;http://amzn.to/2CgKwQR&#34; target=&#34;_blank&#34;&gt;Mantiz&lt;/a&gt; rather than the &lt;a href=&#34;http://amzn.to/2Ci1wpR&#34; target=&#34;_blank&#34;&gt;Vega64&lt;/a&gt; + Mantiz given the Vega64 requires an unsupported/warranty-breaking modification.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Improving Tableau&#39;s Document API</title>
      <link>http://justnumbersandthings.com/post/2017-06-04-improving-tableaus-document-api/</link>
      <pubDate>Sun, 04 Jun 2017 00:00:00 +0000</pubDate>
      <guid>http://justnumbersandthings.com/post/2017-06-04-improving-tableaus-document-api/</guid>
      <description>

&lt;h1 id=&#34;1-what-is-tableau-s-document-api&#34;&gt;1) What is Tableau&amp;rsquo;s Document API?&lt;/h1&gt;

&lt;p&gt;With the release of Tableau 10, Tableau released a &lt;a href=&#34;https://github.com/tableau/document-api-python&#34; target=&#34;_blank&#34;&gt;python utility&lt;/a&gt; called the Tableau Document API (or TDA for short). TDA allows users to easily programmatically modify tableau workbooks. Modifying tableau workbooks without using Tableau Desktop was possible before as tableau files &lt;code&gt;.twb&lt;/code&gt; are actually just xml files. However, manually editing the xml of &lt;code&gt;.twb&lt;/code&gt; files could easily result in a corrupted workbook. Fortunately with the release of this tool it is now much less risky to modify workbooks without using Tableau Desktop.&lt;/p&gt;

&lt;h1 id=&#34;2-using-tableau-s-document-api&#34;&gt;2) Using Tableau&amp;rsquo;s Document API&lt;/h1&gt;

&lt;p&gt;TDA is written in Python so using it is as simple as &lt;code&gt;pip install TableauDocumentApi&lt;/code&gt; and &lt;code&gt;import TableauDocumentApi&lt;/code&gt;. For example if you needed to update the connection strings in a dozen local tableau files you could use the following script to update them all.&lt;/p&gt;

&lt;pre&gt;&lt;code class=&#34;language-python&#34;&gt;import TableauDocumentApi as tda
import glob, os

os.chdir(&amp;quot;my_folder&amp;quot;)
for file in glob.glob(&amp;quot;*.twb&amp;quot;):
  tableau_workbook = tda.Workbook(file)
  for datasource in tableau_workbook.datasources:
    for connection in datasource.connections:
      connection.server = &#39;my-new-host&#39;
tableau_workbook.save()
&lt;/code&gt;&lt;/pre&gt;

&lt;h1 id=&#34;3-tableau-s-query-bands-initial-sql&#34;&gt;3) Tableau&amp;rsquo;s Query Bands &amp;amp; Initial SQL&lt;/h1&gt;

&lt;p&gt;Taking a step away from TDA for a moment Tableau has two less visible but still immensely useful features: &lt;a href=&#34;http://onlinehelp.tableau.com/current/pro/desktop/en-us/examples_teradata.html#initial_sql&#34; target=&#34;_blank&#34;&gt;Initial SQL&lt;/a&gt; &amp;amp; &lt;a href=&#34;http://onlinehelp.tableau.com/current/pro/desktop/en-us/examples_teradata.html#query_band&#34; target=&#34;_blank&#34;&gt;querybands&lt;/a&gt;. I&amp;rsquo;ve personally used these features to stage temporary tables &amp;amp; tag each query (there are so many&amp;hellip;) that tableau runs.&lt;/p&gt;

&lt;h1 id=&#34;4-how-does-this-relate-to-the-tableaudocumentapi&#34;&gt;4) How does this relate to the TableauDocumentAPI&lt;/h1&gt;

&lt;p&gt;The &lt;a href=&#34;https://github.com/t8y8/document-api-python/blob/3a38058f168e29874dc95af4f697888bba71a4fe/tableaudocumentapi/connection.py&#34; target=&#34;_blank&#34;&gt;first version I used of Tableau&amp;rsquo;s Document API&lt;/a&gt; did not support modifying initial sql &amp;amp; query bands. After being faced with modifying thousands of workbooks to use query bands I decided to review the library and see if support could be easily added. Glancing at the source code I could see that each of the properties of a connection was cleanly defined. Using Port as an example I was able to submit a &lt;a href=&#34;https://github.com/tableau/document-api-python/pull/123&#34; target=&#34;_blank&#34;&gt;pull request&lt;/a&gt; implementing the functionality. Fortunately, the Tableau team was very responsive and quickly merged the pull request.&lt;/p&gt;

&lt;h1 id=&#34;5-tying-it-all-together&#34;&gt;5) Tying it all together&lt;/h1&gt;

&lt;p&gt;So now if you find yourself in a situation where you need to modify the query bands or the initial sql in your workbooks you can use the TDA to save yourself some time.&lt;/p&gt;

&lt;p&gt;Update: Looks like Tableau was even kind enough to mention this new feature in the &lt;a href=&#34;https://www.tableau.com/about/blog/2017/2/check-out-new-enterprise-features-tableau-102-66459&#34; target=&#34;_blank&#34;&gt;release notes of 10.2&lt;/a&gt; Search for &amp;ldquo;and with our Document API&amp;rdquo;&lt;/p&gt;

&lt;p&gt;Cheers!&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title> Connecting to Hive with DBeaver using Kerberos Authentication</title>
      <link>http://justnumbersandthings.com/post/2017-05-06-dbeaver-hive/</link>
      <pubDate>Sat, 06 May 2017 00:00:00 +0000</pubDate>
      <guid>http://justnumbersandthings.com/post/2017-05-06-dbeaver-hive/</guid>
      <description>

&lt;h1 id=&#34;0-install-dbeaver&#34;&gt;0) Install DBeaver&lt;/h1&gt;

&lt;p&gt;You can find installation instructions &lt;a href=&#34;http://justnumbersandthings.com/post/2017-03-26-dbeaver-mac/&#34;&gt;here&lt;/a&gt;&lt;/p&gt;

&lt;h1 id=&#34;1-download-the-latest-drivers&#34;&gt;1) Download the latest drivers&lt;/h1&gt;

&lt;p&gt;You can find the latest drivers on the &lt;a href=&#34;https://www.cloudera.com/downloads/connectors/hive/jdbc.html&#34; target=&#34;_blank&#34;&gt;Cloudera website&lt;/a&gt;&lt;/p&gt;

&lt;h1 id=&#34;2-create-a-folder-to-store-the-drivers&#34;&gt;2) Create a folder to store the drivers&lt;/h1&gt;

&lt;p&gt;&lt;code&gt;mkdir ~/.dbeaver-drivers/cloudera-hive/&lt;/code&gt;&lt;/p&gt;

&lt;h1 id=&#34;3-extract-driver-jars-and-move-to-the-folder-we-made-earlier&#34;&gt;3) Extract driver jars and move to the folder we made earlier&lt;/h1&gt;

&lt;p&gt;&lt;img src=&#34;http://justnumbersandthings.com/img/20170506_hive_jars_in_place.png&#34; alt=&#34;jars_in_folder&#34; /&gt;&lt;/p&gt;

&lt;h1 id=&#34;4-create-a-new-driver-in-dbeaver&#34;&gt;4) Create a New Driver in DBeaver&lt;/h1&gt;

&lt;ol&gt;
&lt;li&gt;Navigate to &lt;code&gt;Database &amp;gt; Driver Manager &amp;gt; New&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Add all the files from &lt;code&gt;~/.dbeaver-drivers/cloudera-hive/&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Driver name: &lt;code&gt;Hive-Cloudera&lt;/code&gt; (for labeling only)&lt;/li&gt;
&lt;li&gt;Class name: &lt;code&gt;com.cloudera.hive.jdbc41.HS2Driver&lt;/code&gt; (at the time of this writing)&lt;/li&gt;
&lt;li&gt;Default port: &lt;code&gt;10000&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;URL template: &lt;code&gt;jdbc:hive2://{host}:{port}/{database};AuthMech=1;KrbRealm=FOO.BAR;KrbHostFQDN={server}; KrbServiceName=hive;KrbAuthType=2&lt;/code&gt;

&lt;ul&gt;
&lt;li&gt;Note you need to change &lt;code&gt;FOO.BAR&lt;/code&gt; to match your krb5.conf settings&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h1 id=&#34;5-create-a-new-connection&#34;&gt;5) Create a New Connection&lt;/h1&gt;

&lt;ol&gt;
&lt;li&gt;In the menu bar Navigate to &lt;code&gt;Database &amp;gt; New Connection&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Select Hive-Cloudera&lt;/li&gt;
&lt;li&gt;Fill in the appropriate values for host &amp;amp; database (I set database to default)&lt;/li&gt;
&lt;li&gt;Set server to be your KrbHostFQDN&lt;/li&gt;
&lt;li&gt;Leave your user name &amp;amp; password blank&lt;/li&gt;
&lt;li&gt;Test connection&lt;/li&gt;
&lt;li&gt;Press next, next, &amp;amp; change the name of this connection as you see fit&lt;/li&gt;
&lt;li&gt;Press finish&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Congrats you&amp;rsquo;ve successfully connected to hive using kerberos authentication!&lt;/p&gt;

&lt;h1 id=&#34;6-troubleshooting&#34;&gt;6) Troubleshooting&lt;/h1&gt;

&lt;p&gt;If you are receiving &lt;code&gt;[Cloudera][HiveJDBCDriver](500168) Error creating login context using ticket cache: Unable to obtain Principal Name for authentication&lt;/code&gt; make sure to check the following&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Ensure that you have the latest cryptography libraries installed

&lt;ul&gt;
&lt;li&gt;Java 9 includes these libraries by default&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;li&gt;That you&amp;rsquo;ve configured your &lt;code&gt;/etc/krb5.conf&lt;/code&gt; successfully

&lt;ul&gt;
&lt;li&gt;If you&amp;rsquo;ve done this correctly you should be able to run &lt;code&gt;kinit&lt;/code&gt; in terminal and create a ticket without issue&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;

&lt;li&gt;&lt;p&gt;For Windows adding the following lines to your dbeaver.ini may be necessary as well&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;-Djava.security.krb5.conf=c:\kerberos\krb5.ini&lt;/code&gt;

&lt;ul&gt;
&lt;li&gt;note: this is the windows equivalent of &lt;code&gt;/etc/krb5.conf&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;

&lt;li&gt;&lt;p&gt;&lt;code&gt;-Djava.security.auth.login.config=c:\kerberos\jaas.conf&lt;/code&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;success has also been reported with the following jaas.conf file &amp;amp; keytab usage&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;Client {
com.sun.security.auth.module.Krb5LoginModule required
debug=true
doNotPrompt=true
useKeyTab=true
keyTab=&amp;quot;C:\Users\{user}\krb5cc_{user}&amp;quot;
useTicketCache=true
renewTGT=true
principal=&amp;quot;{user}@FOO.BAR&amp;quot;
;
};
&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;/ol&gt;
</description>
    </item>
    
    <item>
      <title>Installing DBeaver on a Mac</title>
      <link>http://justnumbersandthings.com/post/2017-03-26-dbeaver-mac/</link>
      <pubDate>Sun, 26 Mar 2017 21:35:50 -0800</pubDate>
      <guid>http://justnumbersandthings.com/post/2017-03-26-dbeaver-mac/</guid>
      <description>

&lt;h1 id=&#34;0-what-is-dbeaver-http-dbeaver-jkiss-org&#34;&gt;0) What is &lt;a href=&#34;http://dbeaver.jkiss.org/&#34; target=&#34;_blank&#34;&gt;DBeaver&lt;/a&gt;?&lt;/h1&gt;

&lt;p&gt;Quite simply DBeaver is the best multi-database SQL IDE that I&amp;rsquo;ve used. It supports every JDBC connection that I&amp;rsquo;ve thrown at it and has advanced features for some of the more popular databases such as Mysql &amp;amp; Postgres. Many thanks to &lt;a href=&#34;https://github.com/serge-rider&#34; target=&#34;_blank&#34;&gt;serge-rider&lt;/a&gt; for creating such an awesome tool.&lt;/p&gt;

&lt;h1 id=&#34;1-install-java&#34;&gt;1) Install Java&lt;/h1&gt;

&lt;p&gt;First we need to install java which can be easily by running the following command in terminal (&lt;a href=&#34;https://brew.sh/&#34; target=&#34;_blank&#34;&gt;homebrew required&lt;/a&gt;):&lt;/p&gt;

&lt;pre&gt;&lt;code class=&#34;language-bash&#34;&gt;brew cask install java
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Note: Previous versions of this article instructed an installation of the cask &lt;code&gt;jce-unlimited-strength-policy&lt;/code&gt; but that has been removed as its contents have been incorporated into the cask &lt;code&gt;java&lt;/code&gt; with the release of 9.0&lt;/p&gt;

&lt;h2 id=&#34;notes&#34;&gt;Notes&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;I recommend the newest version of Java as DBeaver regularly depreciates old versions of Java

&lt;ul&gt;
&lt;li&gt;&lt;a href=&#34;https://support.apple.com/kb/DL1572?locale=en_US&#34; target=&#34;_blank&#34;&gt;Java SE6&lt;/a&gt; is not supported&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&#34;2-install-dbeaver&#34;&gt;2) Install DBeaver&lt;/h1&gt;

&lt;ol&gt;
&lt;li&gt;Download the &lt;a href=&#34;http://dbeaver.jkiss.org/download/&#34; target=&#34;_blank&#34;&gt;latest version&lt;/a&gt; | &lt;a href=&#34;https://github.com/serge-rider/dbeaver/releases&#34; target=&#34;_blank&#34;&gt;Github Alternate&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;After downloading drag and drop into your application folder

&lt;ul&gt;
&lt;li&gt;The first time you run the application you may need to right-click on the application and then press open&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Congratulations you&amp;rsquo;ve installed DBeaver!&lt;/p&gt;

&lt;p&gt;Your next step is configuring a connection(s) to your database(s).&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>First Blog With Pelican</title>
      <link>http://justnumbersandthings.com/post/2017-03-03-pelican-intro/</link>
      <pubDate>Fri, 03 Mar 2017 00:00:00 +0000</pubDate>
      <guid>http://justnumbersandthings.com/post/2017-03-03-pelican-intro/</guid>
      <description>

&lt;h2 id=&#34;update-2018-03-18-i-ve-since-moved-to-hugo&#34;&gt;Update 2018-03-18: I&amp;rsquo;ve since moved to Hugo&lt;/h2&gt;

&lt;p&gt;This is my first blog post using Pelican and Markdown. A lot of the content below is to be used as reference mostly for myself and any others who are exploring using Python3, Pelican, &amp;amp; Markdown to create a blog.&lt;/p&gt;

&lt;p&gt;How to get up and running&lt;/p&gt;

&lt;pre&gt;&lt;code class=&#34;language-bash&#34;&gt;mkvirtualenv personal_blog
pip install pelican
pip install markdown
pip install fabric3
pip install ghp-import
pip install webassets
npm install less -g
cd Dropbox/projects/python/personal_blog/
git clone https://github.com/r-richmond/rirchmond.github.io.git
git submodule add https://github.com/textbook/bulrush.git
git submodule add https://github.com/getpelican/pelican-plugins.git
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Running commands using Fab3 which help prepare posts&lt;/p&gt;

&lt;pre&gt;&lt;code class=&#34;language-bash&#34;&gt;fab rebuild
fab preview
fab serve
fab clean
fab gh_pages
fab reserve
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;This &lt;a href=&#34;http://nafiulis.me/making-a-static-blog-with-pelican.html&#34; target=&#34;_blank&#34;&gt;post&lt;/a&gt; was very helpful as well&lt;/p&gt;

&lt;p&gt;I followed this &lt;a href=&#34;http://www.curtismlarson.com/blog/2015/04/12/github-pages-google-domains/&#34; target=&#34;_blank&#34;&gt;post&lt;/a&gt; to setup my custom domain&lt;/p&gt;
</description>
    </item>
    
  </channel>
</rss>