Go and JSON: Tags, Empty Strings and Nil Values

I noticed that many struggled with encoding and decoding JSON in Go. Here I cover the issues developers run into and how to solve them.

Recently, I have been spending a lot of time working with Go. While helping other developers I noticed that many struggled with encoding and decoding JSON. In this guide I cover a lot of the common issues developers run into and how to solve them.

Naming keys

In the following example I will define a struct and then marshal it using the encoding/json package.

package main

import (

type Item struct {
	Foo   bool
	Bar string

func main() {
	itemJson, _ := json.Marshal(&Item{Foo: true, Bar: "Test"})

The code above will output the following string.


The keys Foo and Bar are using the same format as our struct. You may want these to be lowercase or different from your struct. To do that we can use struct tags like so.

type Item struct {
	Foo   bool `json:"foo"`
	Bar string `json:"bar_key"`

By only changing the struct and running the same code again we will get the following output.


Excluding keys

We can exclude the keys by adding a - to our json tag in our struct. Any tags which use a - will be excluded when marshalling a struct. So let’s change one of our tags to - like this:

type Item struct {
	Foo   bool `json:"foo"`
	Bar string `json:"-"`

When we run our code again we will get the following output.


As you can see Bar is now excluded from the marshalled string. This is useful if you have structs that serve multiple purposes such as database queries and api responses.

Optional or empty keys when unmarshalling

In some situations you may send or receive JSON with optional keys which are sometimes present and sometimes not. The issue you may run into is that bools are false, ints are 0 and strings are "" when they are not set. I have seen some developers make huge workarounds to deal with this issue but there is a simple solution: pointers. Let’s take a look at an example without pointers.

package main

import (

type Item struct {
	Foo bool `json:"foo"`
	Bar int  `json:"bar"`

func main() {
	itemJson, _ := json.Marshal(&Item{})
	item := Item{}
	emptyJson := "{}"
	_ = json.Unmarshal([]byte(emptyJson), &item)
	fmt.Println(item.Bar, item.Foo)

In the above example we are doing two things. First, we marshal en empty Item struct without any values. Then we take an empty JSON object and Unmarshal this to a new Item struct and then print the values of Bar and Foo. The following is the output we get from running the above code:

0 false

Even though we did not set any values, Bar has a value of 0 and Foo has a value of false. This can be a problem with optional parameters because we don’t know if the data we received was actually set to a value or omitted. In the same way, we might end up sending values which we did not intend to send. Now, let’s try changing our struct to use pointers instead:

type Item struct {
	Foo *bool `json:"foo"`
	Bar *int  `json:"bar"`

If we run the same code but only adding * we get the following output:

<nil> <nil>

Now we have have nil values since we did not set anything. This allows us to check if a parameter was actually set to a value. For example, we might use the following code to check the value of item.Foo after unmarshalling.

if item.Foo == nil {
	// Foo was not set
} else if *item.Foo {
	// Foo was true
} else if !*item.Foo {
	// Foo was false


I hope this post has helped cover the common issues with Go and JSON. If I missed anything or made a mistake please let me know in the comments. Thank you for reading!

Will Github Copilot help me write code?

I am a developer working mainly with Go and PHP on a daily basis. This past weekend I got invited to the Github Copilot Technical Preview and decided to spend my entire weekend testing it out. I know there is some controversy regarding Copilot and I want to disregard that and focus on the practicality of using Copilot as a tool while typing code.

I was working on an addon for a game I play in my spare time called Elder Scrolls Online. The addon is called Trade Guild Ledger and is available on Github and is written in Lua. In addition to the Lua addon it has a client and server written in Go which extracts item pricing data from the game and uploads it to the server.

Overall Impressions

Github Copilot is a nice addition to my toolset as a developer, it’s like autocomplete when writing emails in Gmail. This is how Gmail auto-completes your sentences:

Gmail auto-complete finishes sentences for you.

Copilot gets rid of the boring, repetitive tasks that I am doing over and over again as a developer. Here is an example of how Copilot auto-completes your code:

Github Copilot finishes code snippets for you as you are programming.

In the same way that Gmail can make typing emails faster, Copilot makes me able to write code faster. Copilot runs as a Visual Studio Code extension and code hints typically show in less than a second. If I am typing code fast I don’t see suggestions but it is when I pause to think that a suggestion typically pops up.

Most of the suggestions I have seen while working normally have been suggestions which are based off of other code in the same file. During regular use I did not see any obvious code which was a verbatim copy from other repositories or anything that resembled sensitive data. There was only one suggestion I noticed which was completely new to my own codebase, shown below.

Github Copilot gives a good suggestion for a function named regionFromIndex. Only typing the name and first parameter Copilot is able to finish the rest of the functions logic automatically.

In this example Copilot gets the names of my structs right and also knows what I want to return. A pretty simple example but it saved me time as I didn’t have to write this simple function.

Bad suggestions

Copilot doesn’t always get the right suggestions as I expected. Just like auto-complete for typing gets the wrong sentences. When this happens I simply ignore the suggestion and keep typing. I think the most annoying suggestions are those for really large code snippets like this example:

Github Copilot gives a bad suggestion which is a verbatim copy of a code snippet in the same file.

The code suggestion above gets cut off after 30 lines or so but it is a copy of another function in the same file. The only difference between the two is that Copilot has replaced all occurrences of “Items” with “Listings” instead.

As a developer you would need a bit of experience to determine of the code being suggested is right or not. The code suggested is also not the best code most of the time. Copilot saves me time where I might otherwise use a search engine to find an answer.

Would I use Github Copilot all the time?


Am I going to use Copilot all the time? No.

For personal projects I will keep using it as much as I can, it definitely helps more than it hurts. For work I will wait a bit and see how things play out as I know that not all of my colleagues are huge fans of Copilot. I hope that Copilot starts discussions that lead to better tools for everyone in the end.

Github Copilot will probably not improve my code or magically replace what I do. However, it can speed up my work by automating repetitive functions which I keep writing over and over again in slightly different variations.

Run your own NameServers with CoreDNS

I have a lot of domains and I wanted an easy way to manage all of them without having to deal with some kind of API from a third party. For learning purposes I decided to give CoreDNS a try. All of my domains are for my own hobby projects and I decided to try running my own nameservers to learn. There are plenty of free options available and it is not recommended to run your own nameservers.

The things I will be using in this guide are mainly:

  • CoreDNS
  • Ansible
  • Hetzner cloud – instances are in Finland, costs around 6 euro per month
  • Namesilo (affiliate link, use code GOPHP to get $1 off ) – for registering our nameservers


I went ahead and deployed 2 Hetzner cloud instances since they are cheap and easy to deploy. I am also based in Europe which puts the NameServers close to me which is good for me, not so good for my US users. You could also use AWS EC2 or Google Cloud Compute instances or anything else you prefer.


I wanted a super simple way to manage my domains so I decided to make use of Ansible to deploy my NameServers. This allows me to keep track of all configurations in one big YAML file. Here is an example of how I have defined my domains.

# roles/common/vars/main.yml
  - domain: 'uberswe.com'
      - '10 in1-smtp.messagingengine.com'
      - '20 in2-smtp.messagingengine.com'
      - 'v=spf1 include:spf.messagingengine.com ?all'
      - domain: 'fm1._domainkey'
          - 'fm1.uberswe.com.dkim.fmhosted.com'
      - domain: 'fm2._domainkey'
          - 'fm2.uberswe.com.dkim.fmhosted.com'
      - domain: 'fm3._domainkey'
          - 'fm3.uberswe.com.dkim.fmhosted.com'
  - # bokföring.xyz
    domain: 'xn--bokfring-q4a.xyz'
  - # bolån.xyz
    domain: 'xn--boln-soa.xyz'

Here we have a variable called domains with a list of objects. Each object needs to specify a domain and the rest is optional.

CoreDNS Configuration

I can use the domains variable to create dns zone files and a CoreDNS configuration file which our dns server will read. We can use a Jinja template to make our configuration file. Below is the template I made.

.:53 {
    forward .

{% for d in domains %}
{{d.domain}} {% if 'sub' in d %}{% for s in d.sub %}{{s.domain}}.{{d.domain}} {% endfor %}{% endif %}{
    file /etc/coredns/zones/{{d.domain}}.db

{% endfor %}

Our nameservers will forward any requests it doesn’t match to as a fallback. This file will take the domains list we defined in our YAML file and loop through each domain object specifying the db file for each domain.

I decided to use files as CoreDNS has a file plugin which seemed easy to use. I would like to try using the redis plugin in the future to load zone data from a redis instance. Below is my Jinja file which takes each object in the YAML file above and uses it to generate a dns zone file.

$ORIGIN {{ item.domain }}.
@	3600 IN	SOA ns1.beubo.com. admin.beubo.com. 2017042745 7200 3600 1209600 3600
	3600 IN NS ns1.beubo.com.
	3600 IN NS ns2.beubo.com.
        IN A
        IN AAAA	2a01:4f9:2a:d09::2
{% if 'mx' in item %}{% for mx in item.mx %}        IN MX  {{mx}}.
{% endfor %}{% endif %}
{% if 'txt' in item %}{% for txt in item.txt %}        IN TXT  "{{txt}}"
{% endfor %}{% endif %}
{% if 'cname' in item %}{% for cname in item.cname %}        IN CNAME  {{cname}}
{% endfor %}{% endif %}

*       IN A
        IN AAAA  2a01:4f9:2a:d09::2

{% if 'sub' in item %}{% for d in item.sub %}
{% if 'ipv4' in d %}{{d.domain}}       IN A  {{d.ipv4}}
{% endif %}
{% if 'ipv6' in d %}{{d.domain}}       IN AAAA  {{d.ipv6}}
{% endif %}
{% if 'mx' in d %}{% for mx in d.mx %}{{d.domain}}        IN MX  {{mx}}.
{% endfor %}{% endif %}
{% if 'txt' in d %}{% for txt in d.txt %}{{d.domain}}        IN TXT  "{{txt}}"
{% endfor %}{% endif %}
{% if 'cname' in d %}{% for cname in d.cname %}{{d.domain}}        IN CNAME  {{cname}}.
{% endfor %}{% endif %}

{% endfor %}{% endif %}

Currently the nameservers, SOA record and IP addresses for the root domains are hardcoded but could easily be replaced with variables. The basic idea here is that I check if an item has a key then I echo the value. I also added the sub item which contains any sub domains so I use a loop for that item. Here is an example zone file which I have generated for my dns server.

$ORIGIN uberswe.com.
@	3600 IN	SOA ns1.beubo.com. admin.beubo.com. 2017042745 7200 3600 1209600 3600
	3600 IN NS ns1.beubo.com.
	3600 IN NS ns2.beubo.com.
        IN A
        IN AAAA	2a01:4f9:2a:d09::2
        IN MX  10 in1-smtp.messagingengine.com.
        IN MX  20 in2-smtp.messagingengine.com.
        IN TXT  "v=spf1 include:spf.messagingengine.com ?all"

*       IN A
        IN AAAA  2a01:4f9:2a:d09::2

fm1._domainkey        IN CNAME  fm1.uberswe.com.dkim.fmhosted.com.

fm2._domainkey        IN CNAME  fm2.uberswe.com.dkim.fmhosted.com.

fm3._domainkey        IN CNAME  fm3.uberswe.com.dkim.fmhosted.com.

We can write an ansible test to ensure that this is working as it should before we try to run this on our nameservers. Below is my test file.

# tests/test.yml
- hosts:
    - ../roles/common/vars/main.yml
    - name: Create a directory
        path: zones
        state: directory
      delegate_to: localhost

    - name: Test dns zone files generation
      template: src=../roles/dnstier/templates/dnszone.db.j2 dest=zones/{{ item.domain }}.db
      with_items: "{{ domains }}"
    - name: Create a directory
        path: files
        state: directory
      delegate_to: localhost

    - name: Test Corefile for coredns generation
      template: src=../roles/dnstier/templates/Corefile.j2 dest=files/Corefile

You can run this with ansible-playbook tests/dnszones.yml and it should make a folder called zones which will contain a .db file for each domain. A files directory with the Corefile will also be created.

Deployment Configuration

To setup and configure CoreDNS I found ansible role which was already ready to use at https://github.com/cloudalchemy/ansible-coredns. We can install this role using the command ansible-galaxy install cloudalchemy.coredns.

We need to generate the zone files locally so I created a role called dnstier which will prepare these files using the following task file.

# file: roles/dnstier/tasks/main.yml
- name: Create a directory
    path: /tmp/zones
    state: directory
  delegate_to: localhost

- name: dns zone files generation
  template: src=../templates/dnszone.db.j2 dest=/tmp/zones/{{ item.domain }}.db
  with_items: "{{ domains }}"
  delegate_to: localhost

Now we can set up our main playbook which will run both of these roles.

# file: dnsservers.yml
- hosts: dnsservers
    - dnstier
    - cloudalchemy.coredns

We also need to define our variables for the cloudalchemy.coredns role. This is how I defined them.

# file: group_vars/all.yml

coredns_config_file: "roles/dnstier/templates/Corefile.j2"

  - "/tmp/zones/*"

coredns_system_group: "coredns"
coredns_system_user: "{{ coredns_system_group }}"

The variables I have defined here come from the readme of the coredns role we installed with galaxy. The config file is the path to the template file we defined earlier and the zone file path is the same as we defined in our dnstier task. I also have a hosts file defined like so.

# host_vars/all.yml

Here beubo and ns2beubo are the host names I have defined in my ssh config for my two nameservers. We could add more hosts here and ansible will deploy this same setup to each one.

Deploying the Nameservers

Now we can finally go ahead and deploy using the command ansible-playbook dnsservers.yml which will:

  • Run the dnstier role, generating the zone files.
  • Run the cloudalchemy.coredns role which installs our coredns server .

After the deployment is done we can check that it is working by specifying the IP of each dns server with a @ symbol in the dig command. You can run this for example to query my domain and server dig @ uberswe.com

We can now register our nameservers. I am doing this with Namesilo but most registrars should provide an easy way of doing this. I will enter ns1 and ns2 as the prefixes and enter a ipv4 and ipv6 address for each nameserver. You can find the ip addresses on your Hetzner cloud instance page.

Namesilo page to add a new registered nameserver.

Once you have done that and waited some time for the dns to propagate you can query a domain using your server. We can replace the ip in our previous dig query with the domain like so dig @ns1.beubo.com uberswe.com.

If that is working like it should, you should then be ready to point your domains to your new nameservers. Feel free to leave a comment if you have questions, see something I did wrong or just want to wish me a good day.

If this stuff interests you, you may be interested in some relevant topics I found while setting up my nameservers:

Minimalism in Web Development

As I spend more and more time working with bloated frameworks in web development I am starting to miss the minimalism I saw when I got started. I wish websites didn’t need to load megabytes of data to display simple pages. I believe a lot of websites do not benefit at all from frameworks like React, Vue and Angular. Some have told me that these pages just need to be optimized but as far as I can tell this just spreads the loading of files out over time, you will still end up with a site of several megabytes after navigating pages.

How do we define minimalism?

I believe we need less javascript and css for the web. Things can still look great and be done quickly without a bunch of frameworks. I believe this is what we like to call minimalism in web development. Looking at Wikipedia I found a good definition for minimalism and it’s defined like so:

In software and user interface design, minimalism describes the usage of fewer design elements, flat design, fewer options and features, and tendencially less occupied screen space. 

This definition fits well and I believe it’s good to follow this principle. However, I have seen many minimalist user interfaces that look great but take ages to load and use megabytes of data. If we try to use minimalism in web design shouldn’t we also use minimalism when it comes to the underlying code? I found an article called My Minimalist Approach To Software Engineering where the author touches on this topic. Libraries and frameworks can be good but there is a lot we can do in web development without them. In this article and hopefully future articles I would like to explore what we can do with minimalist web development.

In addition to what I have written here there is an article called A Short Guide To Minimal Web Development which I found while researching this topic. I think it outlines a lot of the things for frontend development which I would like to cover but I also want to apply the principle to backend development. Frameworks and libraries have their uses but they are not needed for most web applications.

The idea, however, is that we can only tell what’s really needed once we can tell what’s really important, and why it’s really important.

Jens Oliver Meiert (https://meiert.com/en/blog/minimal-web-development/)

When do we need a backend?

You may need a backend to a web application when you have dynamic content. Now, there is no simple definition for dynamic content. Basically, if you have any form of user authentication or storage shared between users you will need a backend. A lot of things such as blogs, games, live chats and more may have dynamic content but do not need any backend to perform their function if authentication is not needed. Most sites will fall into the first category of authentication but there are some sites out there which need a backend but do not require authentication. These types of sites may exists to provide simple functions like uploading and sharing files or tracking and storing user behaviour.

Many sites like blogs or even news sites will do fine with basic HTML files with no extra code. However, for convenience some people do like to use a backend such as WordPress. WordPress lets you easily create new posts and pages. This does however come with the added cost of complexity and performance. HTML files tend to load quickly and there is very little that can go wrong. Once you add a backend with authentication there may be hundreds of checks that execute on every page load. You also have to maintain a backend, keep it updated and ensure that malicious users don’t gain access to it.

When making a new site you should always begin by asking if you really need a backend.

When is a backend framework needed?

A backend framework is never really needed. But a framework can help speed things up and provide a ready made structure. It’s similar to building a house, you build everything from scratch or use prefabricated houses where all the pieces are already made.

A framework therefore helps standardise things so that it is easier for multiple developers to work on the same project. However, there should always be some consideration as to if a framework is really needed. If you’re building a landing page you don’t need frameworks. You could probably use a simple PHP file for this instead of a full framework like Laravel. Using a framework will add overhead and complexity to the application, especially if it’s small. In the end you are trading development speed for flexibility, stability, performance and security.

When is a frontend framework needed?

Frontend frameworks like React, Angular and Vue have become very popular in the last years. I think these frameworks are needed even less than backend frameworks. Most websites do not need any javascript at all to provide their core functionality. The backend can generate html pages with any needed data and this can be styled with a bit of css.

One argument I have heard is that these frameworks allow you to load a page once and then only make requests for the data that you need. This is referred to as a single page application and I can see how it provides benefits for large applications with lots of data. However, most websites do not load large sets of data in this way and can provide pages that are small in size and load quickly. Many single page applications load so many different libraries that they often reach over a megabyte in size which is huge.

You end up trading a slow initial load for many smaller pages that load quickly. It’s also tends to become impossible to keep track of all the libraries that these frontend applications use. There have been several recent incidents where a small package had a bug or security issue which affected millions of applications. Frontend frameworks should be avoided unless they provide substantial benefits. Frontend frameworks might allow you to get something produced more quickly but once again you trade development speed for flexibility, stability, performance and security.


As I gain more and more experience in web development I am starting to miss the simplicity and speed of the old days. I think we need to find a way back to the simple application. I am personally starting to simplify the websites I run.

As I write this I am using WordPress to manage my posts. However, I am working on a new backend called Beubo. With Beubo I want to simplify the backend to something very basic and rethink how the backend should work. I want to write guides that teach minimalism in web development from the very start. Too many web developers start with frameworks and never learn the simplicity of vanilla CSS, JS and HTML.

I think there is more and more push for simplicity and minimalism in web development. This is as companies become more and more liable for the security of their applications. For a long time there were no real consequences of running insecure web applications. Focus has shifted towards development speed. However, with the rise of GDPR and similar laws we will see more companies looking to simplify. To reduce the amount of libraries and frameworks that are used in order to keep applications more secure and stable.

I would love to hear what you guys think about minimalism in web development. Please leave your comment below.

How Switching Domain Registrars Saved Me $300+/Year

When registering a domain it’s typically easy to find domain registrars with good deals, perhaps just $1 to get any domain name you want. That’s a trick to reel you in in the hopes that you will keep that domain for many years without switching registrar. The renewal fees and any additional offerings such as hosting and email is where most domain registrars earn the most money. Today, however, I want to simply take a look at renewal fees and how much extra we spend on renewals. If you are a company or someone like me with 100+ domains these renewal fees can get very expensive. Here I will show how much you can actually save on renewal fees by switching domain registrars and how I went about figuring this out.

Continue reading “How Switching Domain Registrars Saved Me $300+/Year”

Customer Feedback via SMS using Go and Slack

Using an SMS Gateway called 46elks and some Go we can automate a customer feedback process which is usually done manually by calling a customer. This was an idea I had when I worked at a large sales company in Sweden a few years ago and my task was to find things that could be automated to save money and speed things up. To keep track of feedback and manually handle negative comments I also piped the messages to Slack which we will be doing in this guide as a bonus at the end. The following is a how to guide describing how you can create this yourself. You can find the full source code on Github.

SMS Gateway

The first thing we need to is find a SMS Gateway for sending our text messages to our customers. I met 46elks at HackForSweden this year and it seems to perform really well when I used them for this guide. However, you could go with any SMS Gateway and some of the more popular options are Twilio, Amazon or ClockworkSMS. Most gateways will provide similar APIs so it shouldn’t be too hard to replace my 46elks implementation with another provider.

Sending texts and receiving customer feedback

Continue reading “Customer Feedback via SMS using Go and Slack”

How To Run A Go Web Server On AWS

Amazon Web Services (AWS) offers free tier web servers where you can run simple go programs for API backends or websites. In this guide we will make a simple web server using a go program. This guide can also be used as a reference to set up AWS instances for larger go services.

To begin you will need to set up a new instance on AWS. To do this simply login to your AWS account and click on Services at the top of the screen and then on EC2. EC2 should be the first option under the Compute category.

Click on Instances on the left hand side and then in the top left there should be a large button called Launch Instance which will start a creation wizard. I typically search for Ubuntu and pick 18.04 as shown in the screenshot below.

Continue reading “How To Run A Go Web Server On AWS”

How to use Pi-hole to block all ads on your local network

Recently I decided to set up a Pi-hole installation on my local network. Initially I was going to use a Raspberry Pi 3 B+ but the company I work for was replacing all the PCs and I was able to get my hands on one of them.

Pi-hole is a simple tool which can be configured to handle the DNS of all the devices on a network. It can be installed on a linux environment and is also available as a docker image. Basically it will check any DNS queries against various blacklists and block queries that match ad networks or tracking domains. The end result is that you get a browsing experience without advertisements and user tracking. Pi-hole is configurable and you can whitelist domains or even devices on the network if you would like. The admin interface shown in the image of this post shows the dashboard which gives a nice overview of queries that go through Pi-hole.

Continue reading “How to use Pi-hole to block all ads on your local network”

How to Install Go 1.12 on Ubuntu 18

This guide is also available as a video from Youtube.

Begin by running the following command to add the golang backports ppa:

sudo add-apt-repository ppa:longsleep/golang-backports

If you get an error such as “add-apt-repository command not found” then you need to run the following command to install add-apt-repository:

Continue reading “How to Install Go 1.12 on Ubuntu 18”

Create a Public Slack Community with Go

I recently stumbled across a tweet by Melvin Davis on Twitter where he mentioned a simple tool that he had created using Go that instantly sends a Slack invite when an email is entered. This allows us to make a public slack community that anyone can join.

Check-out the auto invite sender to join #slack written in #Golang https://t.co/fwcurIsZ3w This tool is extremely helpful when you maintain an community slack channel— Melvin Davis (@melvinodsa) January 2, 2019

Continue reading “Create a Public Slack Community with Go”