burger-stack

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

(★ 20)

Presentation for "The Burger Stack"

burger-stack Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

The Burger Stack: Simplifying Web Application Development
=========================================================

### Target Audience

Intermediate developers looking to simplify development.

### Introduction (5 sentences)

Gophers love to talk about how great Go is. And it is! I've used it to write
databases and distributed systems and command line tools. It's my favorite
language. But ever since I started writing Go there's been one area that
everyone advises against using Go: web development. And that's terrible. Web
applications are an enormous part of our industry!

Additionally, web development hasn't seen limited improvement over the years.
We started with the LAMP stack 20 years ago where we used slow languages like
PHP or Perl to connect to a difficult to configure MySQL server. 10 years later
we saw Rails where we used Ruby, a slow language, to connect to complex SQL
databases. Recently we've see the MEAN stack where we write JavaScript on the
backend to connect to MongoDB. It's almost like we've gone backwards in our
industry.

However, all is not lost. If we consider some recent developments in our
industry over the past few years, Go begins to stand out as a serious contender
within the web application space.

### Single Page Applications

One big flaw with Go is that it lacks a strong templating language. There is
the `template` package but it's difficult to use at scale for a large
application. However, in recent years there has been enormous adoption for
Single Page Applications (SPAs). We've seen several technologies come along such
as AngularJS, Backbone.js, & Ember.js but none of these have seen the crazy
adoption that Facebook's React has.

React does a lot of awesome UI stuff and if this was a JavaScript conference
then I'd talk about it more. However, it's important to note that React renders
client side so it only needs to consume an API. No templating is needed on the
server side.

### SQL Alternatives

Another flaw that Go has is that working with SQL databases is not much fun.
Rails has a pretty ORM built-in but you still need to know enough to optimize
things like N+1 queries. You also need to configure and maintain your database
server. Honestly, it feels like we adopted relational databases 40 years ago
as a standard and we've been doing whatever we can to avoid SQL ever since.

However, there is a much better way to avoid SQL, relational databases, and set
theory. Simply don't use them. Go excels at working with byte slices so instead
of mapping objects to relations, you can map your objects to bytes.

Now if you think that mapping to bytes is complex, don't worry, this is a
problem that computer scientists have devoted years of research to and it's
called "serialization". There are so many serialization libraries to choose
from: Protocol Buffers, FlatBuffers, Avro, MessagePack, JSON. This serialization
is essentially the schema for your data.

Once we're mapping to bytes, we don't need the high level features of most
databases. We can get rid of SQL. Since we have no SQL we can get rid of the
query planner and query cache. Let's go one step further and get rid of the
server altogether and use an embedded database. At this point we have the
simplest database possible: an embedded key/value store.

There are several key/value stores available but we want something in pure Go,
strong transactional support, and read optimized so we're going to talk about
the most amazing database ever created: BoltDB. Disclaimer: I'm the author of
BoltDB so I'm a little biased.

Bolt, Go, React: the BGR stack. That's all we need to build an extremely fast,
easy to develop, easy to deploy web application. Many of you are probably
familiar with builing APIs with Go and I'm not going to give a React talk at a
Go conference so let's look how you use Bolt for everyone's favorite web app
construct: CRUD.

### Domain Types & Serialization

The first thing you need in any web app is a way to manage users so let's build
a `UserStore`. We'll keep it simple and our user will have two fields: an ID
and a username.

We'll need to serialize this object too so let's use Protocol Buffers to do
that. I like protobufs because it's really fast and it has versioning built-in.
Here's our protobuf definition for our `User` type. Now we can add a
"generate" line in our source and it'll automatically regenerate whenever we
call `go generate`. I also like to put this generated code in my `internal`
package so it's hidden from the rest of my application and godoc.

Next, we'll implement the little known `BinaryMarshaler` and `BinaryUnmarshaler`
interfaces. To marshal, we populate our protobuf object and marshal it. To
unmarshal, we unmarshal our protobuf object and copy from it.

### Opening & initializing

Bolt databases are extremely simple. It's just a single file. You can open it
using `bolt.Open()` similar to how `os.Open()` works. Once our database is open,
we'll need a place to store our user data and Bolt has something called a
`Bucket` for this. A bucket is essentially a persisted map that maps unique
byte slice keys to byte slice values. I use buckets analogously to tables in a
relational database.

We can ensure that we have a "Users" bucket created by starting a transaction
and calling `CreateBucketIfNotExists()`.

Transactions are an important part of Bolt. There are two kinds: read-only and 
read-write. They are full ACID transactions with serializable isolation. That
means that they will have a snapshot view of the data from when the transaction
started. You can have an unlimited number of simultaneous read-only transactions
but only one read-write transaction can operate at a time.

We start a transaction using `Begin()` and the only parameter is whether the
transaction is writable. Make sure you ALWAYS defer rollback on your
transactions! Leaving a transaction open in the event of an early return or a
recovered panic will cause problems.

### Creating a user

Let's add a new user. Let's walk through the code to see how this is done.
First we start a read-write transaction. Next we'll grab our "Users" bucket.
One each bucket there exists an autoincrementing sequence similar. We can grab
a new ID for our user from there.

Since we've already implemented `BinaryMarshaler`, we can easily convert our
user to bytes. Then we'll map our ID to our encoded bytes using `Put()`.
Finally, we'll commit the transaction and return an error if it failed for any
reason. Again, these are true ACID transactions so any operations that occurred
will be rolled back if anything fails.

You'll notice in the `Put()` line I use a helper function called `itob()`. This
is short for integer to bytes. I use this a lot and it saves me 3 lines over
and over again. It also ensures that you're always using big endian encoding.
Endianness is simply how you encode an integer to bytes and we always use
big endian in Bolt because it makes our integer keys sortable.

### Reading a user

Once you have users in your system, you need to get them out. Let's walk through
it. Again, we start a transaction. This time it is read-only. It's important to
use read-only transactions for read-only operations so that it can scale.
Read-write transactions can only operate one at a time. Next we'll pull out our
bytes using the `Get()` method on our bucket. If there is no value then the
key doesn't exist and we can return a `nil` user. But if we get a value then
decode it and return it. The defer rollback will be called and close out the
transaction automatically.

If you want to read all our users you can use a cursor on your bucket. A cursor
is simply a way to iterate over a bucket. We'll start the transaction the same
as before, then grab a cursor, and then we can traverse the whole bucket in a
simple `for` loop. This calls `First()` to move the cursor to the beginning and
then calls `Next()` until there are no more keys. We'll unmarshal and append on
our slice for each user and return.

### Updating a user

We've covered the "C" and the "R" of CRUD, next is update. Update is a bit like
read and create put together. This time we'll use a different method in Bolt for
running transactions called `Update()`. This will run a function in the context
of a transaction. If the function returns an error, everything will rollback.
Otherwise it will commit. This can be really helpful if our operation only
returns an error.

We'll also group some of the operations in if/else statements so it's easier to
see related operations. First we grab the bucket and the value, just like in our
read operation. If it doesn't exist we'll return an error. Otherwise we'll
unmarshal the object. Once we have our object we can update it's state. Now with
the new state we'll marshal and overwrite our previous value. Finally, returning
`nil` at the end will cause our transaction to commit.

### Deleting a user.

Finally we come to the "D" of CRUD. This operation is incredibly simple. We
can use the `Update()` helper and simply delete the key from our bucket.

And that's it! Full set of CRUD operations in a handful of lines of code. And
the best part is that it's all Go.

### BoltDB in Practice

There's a lot more to applications than just writing code. There's testing,
deploying, and maintenance. Testing is quite simple. It's an embedded database
and a single file so you can simply create a temp file for your test and delete
it when the test is done. That also means that you can run your tests in
parallel.

When you deploy BoltDB, it's advised that you use SSDs. Bolt doesn't use
compactions so there's a lot of random writes. Another gotcha is that Bolt
returns byte slices that directly point to an mmap so these slices are only
valid until you close a transaction. One huge benefit to this mmap, though, is
that the data is in the OS page cache so it'll stay there even if you restart
your application. You can use graceful restarts to have zero downtime.

One big question about the maintenance side is how do you backup your data or
handle catastrophic hardware failures? Bolt transactions implement the
`io.WriterTo` interface so you can actually stream a full, consistent copy of
your database to any writer. My personal favorite is adding an HTTP endpoint
so that I can use `curl` as my backup tool.

This does leave a window for data loss between snapshots. There is async
streaming replication support coming in the future so that you can have a
failover server and you can minimize that data loss window to only a few
seconds. However, that is still in early stages. In the meantime you'll need
to evaluate what window is acceptable. An SSD can copy out your database at 
3 - 5 seconds per GB so it may be acceptable to take a snapshot hourly or even
every 10 minutes depending on the size of your database. Sharding your database
can also help keep it small.

Performance is always a question that comes up with any database. Most people
want the "fastest" database but it's difficult to evaluate speed without your
specific workload. Also, most people don't need all the performance they think
they do. Bolt is read optimized which is a good fit for most web applications.
I generally give people these ballpark numbers: 2K random writes/sec, or up to
450,000 writes/sec if you can sort and batch your writes. For reads, it really
depends on if your data is hot in memory. Many applications can fit their
working set in memory -- especially with

### Conclusion (5 sentences)

I believe it's time to move past the slow, complex stacks that permeate through
our industry and adopt something simpler. I've shown how the BGR stack provides
a simple, fast environment for not only developing applications but also
deploying and scaling them. I'm excited to see the future of Go not just in the
distributed systems and low-level services but also in the huge space of web
development. Please, give the BGR stack a try on your next project!

### Key Takeaways (bullet points)

- LAMP & Rails provided great ORM and HTML templating tools
- MEAN tried to modernize with MongoDB & JavaScript but who wants to write JS server side?
- React has seen huge adoption so HTML templating is no longer needed
- BoltDB avoids the headaches of SQL & remote servers so no ORM needed
  - Vertical scaling works for a long time
- How to get started with BoltDB, sample CRUD operations
- BoltDB in production

burger-stack

burger-stack Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

burger-stack

burger-stack Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README