Go and JSON: Tags, Empty Strings and Nil Values

I noticed that many struggled with encoding and decoding JSON in Go. Here I cover the issues developers run into and how to solve them.

Recently, I have been spending a lot of time working with Go. While helping other developers I noticed that many struggled with encoding and decoding JSON. In this guide I cover a lot of the common issues developers run into and how to solve them.

Naming keys

In the following example I will define a struct and then marshal it using the encoding/json package.

package main

import (
	"encoding/json"
	"fmt"
)

type Item struct {
	Foo   bool
	Bar string
}

func main() {
	itemJson, _ := json.Marshal(&Item{Foo: true, Bar: "Test"})
	fmt.Println(string(itemJson))
}

The code above will output the following string.

{"Foo":true,"Bar":"Test"}

The keys Foo and Bar are using the same format as our struct. You may want these to be lowercase or different from your struct. To do that we can use struct tags like so.

type Item struct {
	Foo   bool `json:"foo"`
	Bar string `json:"bar_key"`
}

By only changing the struct and running the same code again we will get the following output.

{"foo":true,"bar_key":"Test"}

Excluding keys

We can exclude the keys by adding a - to our json tag in our struct. Any tags which use a - will be excluded when marshalling a struct. So let’s change one of our tags to - like this:

type Item struct {
	Foo   bool `json:"foo"`
	Bar string `json:"-"`
}

When we run our code again we will get the following output.

{"foo":true}

As you can see Bar is now excluded from the marshalled string. This is useful if you have structs that serve multiple purposes such as database queries and api responses.

Optional or empty keys when unmarshalling

In some situations you may send or receive JSON with optional keys which are sometimes present and sometimes not. The issue you may run into is that bools are false, ints are 0 and strings are "" when they are not set. I have seen some developers make huge workarounds to deal with this issue but there is a simple solution: pointers. Let’s take a look at an example without pointers.

package main

import (
	"encoding/json"
	"fmt"
)

type Item struct {
	Foo bool `json:"foo"`
	Bar int  `json:"bar"`
}

func main() {
	itemJson, _ := json.Marshal(&Item{})
	fmt.Println(string(itemJson))
	item := Item{}
	emptyJson := "{}"
	_ = json.Unmarshal([]byte(emptyJson), &item)
	fmt.Println(item.Bar, item.Foo)
}

In the above example we are doing two things. First, we marshal en empty Item struct without any values. Then we take an empty JSON object and Unmarshal this to a new Item struct and then print the values of Bar and Foo. The following is the output we get from running the above code:

{"foo":false,"bar":0}
0 false

Even though we did not set any values, Bar has a value of 0 and Foo has a value of false. This can be a problem with optional parameters because we don’t know if the data we received was actually set to a value or omitted. In the same way, we might end up sending values which we did not intend to send. Now, let’s try changing our struct to use pointers instead:

type Item struct {
	Foo *bool `json:"foo"`
	Bar *int  `json:"bar"`
}

If we run the same code but only adding * we get the following output:

{"foo":null,"bar":null}
<nil> <nil>

Now we have have nil values since we did not set anything. This allows us to check if a parameter was actually set to a value. For example, we might use the following code to check the value of item.Foo after unmarshalling.

if item.Foo == nil {
	// Foo was not set
} else if *item.Foo {
	// Foo was true
} else if !*item.Foo {
	// Foo was false
}

Conclusion

I hope this post has helped cover the common issues with Go and JSON. If I missed anything or made a mistake please let me know in the comments. Thank you for reading!

Will Github Copilot help me write code?

I am a developer working mainly with Go and PHP on a daily basis. This past weekend I got invited to the Github Copilot Technical Preview and decided to spend my entire weekend testing it out. I know there is some controversy regarding Copilot and I want to disregard that and focus on the practicality of using Copilot as a tool while typing code.

I was working on an addon for a game I play in my spare time called Elder Scrolls Online. The addon is called Trade Guild Ledger and is available on Github and is written in Lua. In addition to the Lua addon it has a client and server written in Go which extracts item pricing data from the game and uploads it to the server.

Overall Impressions

Github Copilot is a nice addition to my toolset as a developer, it’s like autocomplete when writing emails in Gmail. This is how Gmail auto-completes your sentences:

Gmail auto-complete finishes sentences for you.

Copilot gets rid of the boring, repetitive tasks that I am doing over and over again as a developer. Here is an example of how Copilot auto-completes your code:

Github Copilot finishes code snippets for you as you are programming.

In the same way that Gmail can make typing emails faster, Copilot makes me able to write code faster. Copilot runs as a Visual Studio Code extension and code hints typically show in less than a second. If I am typing code fast I don’t see suggestions but it is when I pause to think that a suggestion typically pops up.

Most of the suggestions I have seen while working normally have been suggestions which are based off of other code in the same file. During regular use I did not see any obvious code which was a verbatim copy from other repositories or anything that resembled sensitive data. There was only one suggestion I noticed which was completely new to my own codebase, shown below.

Github Copilot gives a good suggestion for a function named regionFromIndex. Only typing the name and first parameter Copilot is able to finish the rest of the functions logic automatically.

In this example Copilot gets the names of my structs right and also knows what I want to return. A pretty simple example but it saved me time as I didn’t have to write this simple function.

Bad suggestions

Copilot doesn’t always get the right suggestions as I expected. Just like auto-complete for typing gets the wrong sentences. When this happens I simply ignore the suggestion and keep typing. I think the most annoying suggestions are those for really large code snippets like this example:

Github Copilot gives a bad suggestion which is a verbatim copy of a code snippet in the same file.

The code suggestion above gets cut off after 30 lines or so but it is a copy of another function in the same file. The only difference between the two is that Copilot has replaced all occurrences of “Items” with “Listings” instead.

As a developer you would need a bit of experience to determine of the code being suggested is right or not. The code suggested is also not the best code most of the time. Copilot saves me time where I might otherwise use a search engine to find an answer.

Would I use Github Copilot all the time?

Yes.

Am I going to use Copilot all the time? No.

For personal projects I will keep using it as much as I can, it definitely helps more than it hurts. For work I will wait a bit and see how things play out as I know that not all of my colleagues are huge fans of Copilot. I hope that Copilot starts discussions that lead to better tools for everyone in the end.

Github Copilot will probably not improve my code or magically replace what I do. However, it can speed up my work by automating repetitive functions which I keep writing over and over again in slightly different variations.

How Switching Domain Registrars Saved Me $300+/Year

When registering a domain it’s typically easy to find domain registrars with good deals, perhaps just $1 to get any domain name you want. That’s a trick to reel you in in the hopes that you will keep that domain for many years without switching registrar. The renewal fees and any additional offerings such as hosting and email is where most domain registrars earn the most money. Today, however, I want to simply take a look at renewal fees and how much extra we spend on renewals. If you are a company or someone like me with 100+ domains these renewal fees can get very expensive. Here I will show how much you can actually save on renewal fees by switching domain registrars and how I went about figuring this out.

Continue reading “How Switching Domain Registrars Saved Me $300+/Year”

Customer Feedback via SMS using Go and Slack

Using an SMS Gateway called 46elks and some Go we can automate a customer feedback process which is usually done manually by calling a customer. This was an idea I had when I worked at a large sales company in Sweden a few years ago and my task was to find things that could be automated to save money and speed things up. To keep track of feedback and manually handle negative comments I also piped the messages to Slack which we will be doing in this guide as a bonus at the end. The following is a how to guide describing how you can create this yourself. You can find the full source code on Github.

SMS Gateway

The first thing we need to is find a SMS Gateway for sending our text messages to our customers. I met 46elks at HackForSweden this year and it seems to perform really well when I used them for this guide. However, you could go with any SMS Gateway and some of the more popular options are Twilio, Amazon or ClockworkSMS. Most gateways will provide similar APIs so it shouldn’t be too hard to replace my 46elks implementation with another provider.

Sending texts and receiving customer feedback

Continue reading “Customer Feedback via SMS using Go and Slack”

How To Run A Go Web Server On AWS

Amazon Web Services (AWS) offers free tier web servers where you can run simple go programs for API backends or websites. In this guide we will make a simple web server using a go program. This guide can also be used as a reference to set up AWS instances for larger go services.

To begin you will need to set up a new instance on AWS. To do this simply login to your AWS account and click on Services at the top of the screen and then on EC2. EC2 should be the first option under the Compute category.

Click on Instances on the left hand side and then in the top left there should be a large button called Launch Instance which will start a creation wizard. I typically search for Ubuntu and pick 18.04 as shown in the screenshot below.

Continue reading “How To Run A Go Web Server On AWS”

How to use Pi-hole to block all ads on your local network

Recently I decided to set up a Pi-hole installation on my local network. Initially I was going to use a Raspberry Pi 3 B+ but the company I work for was replacing all the PCs and I was able to get my hands on one of them.

Pi-hole is a simple tool which can be configured to handle the DNS of all the devices on a network. It can be installed on a linux environment and is also available as a docker image. Basically it will check any DNS queries against various blacklists and block queries that match ad networks or tracking domains. The end result is that you get a browsing experience without advertisements and user tracking. Pi-hole is configurable and you can whitelist domains or even devices on the network if you would like. The admin interface shown in the image of this post shows the dashboard which gives a nice overview of queries that go through Pi-hole.

Continue reading “How to use Pi-hole to block all ads on your local network”

How to Install Go 1.12 on Ubuntu 18

This guide is also available as a video from Youtube.

Begin by running the following command to add the golang backports ppa:

sudo add-apt-repository ppa:longsleep/golang-backports

If you get an error such as “add-apt-repository command not found” then you need to run the following command to install add-apt-repository:

Continue reading “How to Install Go 1.12 on Ubuntu 18”

Create a Public Slack Community with Go

I recently stumbled across a tweet by Melvin Davis on Twitter where he mentioned a simple tool that he had created using Go that instantly sends a Slack invite when an email is entered. This allows us to make a public slack community that anyone can join.

Check-out the auto invite sender to join #slack written in #Golang https://t.co/fwcurIsZ3w This tool is extremely helpful when you maintain an community slack channel— Melvin Davis (@melvinodsa) January 2, 2019

Continue reading “Create a Public Slack Community with Go”

Developing Slack Bots with Go: Part 1

In my previous post I wrote a guide on how to post a simple Slack message using a bot with PHP. Now we will do the same thing but instead with Go. I have also decided to send a json request instead of a form-url encoded request like we did with PHP. There are some differences, let’s take a look.

Continue reading “Developing Slack Bots with Go: Part 1”

Developing Slack Bots with PHP: Part 1

I have done a lot of work with slack bots in the past both professionally and in my spare time. In the past I build a plugin for WHMCS called WHMCS Slack which was originally a paid addon but is now open source and available for free on Github. This post is a first part in a series of posts i would like to write about both Go and PHP and how we can use these programming languages to create useful bots for Slack.

Continue reading “Developing Slack Bots with PHP: Part 1”