You don’t always need a database
As we begin a new project, we quickly start thinking about what kind of database we are going to use. Options like Oracle, SqlServer, Mysql or even a NoSql database like Mongo or Cassandra always pop up like a good choice.
However, such databases each occupy tens or hundreds of megabytes (gigas?) on disk, besides the fact of another installation, deploy, support, more costs a lot of our time.
Is it worth it?
For most of the time our applications will be storing just a small amount of data. Even thousands of users, clients and products would be taking just some megabytes of your disk. Is it worth it to install a whole database (on another server) just for that? And what if you still don’t know the schema you will end up using and you still have to change the tables all the time?
I’ve been thinking about that for quite some time and one day I found this article by Rob Conery talking about lists being persisted by a json store, with all linq goodness, speed and simplicity. I went crazy about it and started using the library right away in a small project of mine.
I really liked the framework, but very quickly a lot of people jumped in and the library lost its simplicity, stability and you could never know which repository or nuget were the official one. It’s not even on Github anymore :(
Biggy became too big, so I created Mini-Biggy
I started from scratch using the same concept, but adding a lot of features and taking care for not adding complexity. The goal was to add a Nuget package to your project and start coding. Simple, awesome, fast!
So how it works?
Create your project and install MiniBiggy using Nuget:
Install-Package mini-biggy
Create a class called Tweet.
|
1 2 3 4 |
public class Tweet { public int UserId {get;set;} public string Message {get;set;} } |
The code above will create a file called tweets.data with our tweet list serialized. Call Save() or SaveAsync() to persist the list.
|
1 2 3 4 5 6 |
var t = new Tweet(); var list = CreateList<Tweet>.SavingAt("tweets.data") .UsingJsonSerializer() .SavingWhenRequested(); list.Add(t); await list.SaveAsync(); |
And you are all set.
Loading them later
Every time you create a list of some type, it will load all saved objects of that type:
|
1 2 3 4 |
var list = CreateList<Tweet>.SavingAt("tweets.data") .UsingJsonSerializer() .SavingWhenRequested(); var count = list.Count(); //equals 1 |
Linq Goodness
Remember, your list is entirely in memory, so any linq operator will work just fine:
|
1 2 3 |
biggyList.Where(x => x.Username == "admin" && x.Message.ToLower().Contains("urgent")) .OrderBy(x => x.DateTime) .FirstOrDefault(); |
And it’s so fast! Find something in a list of 1,000,000 takes only some milliseconds. In-memory lists are awesome!
Serializers
Mini-biggy comes with 2 serializers:
- JsonSerializer
- PrettyJsonSerializer (the json is indented)
You can change that behavior every time you load your list.
Saving Modes
You can follow 3 strategies to save your list using mini-biggy:
Manual Save
The list will be saved only when Save() or SaveAsync() is called. This is the default behavior.
|
1 2 3 4 5 6 |
var t = new Tweet(); var list = CreateList<Tweet>.SavingAt("tweets.data") .UsingJsonSerializer() .SavingWhenRequested(); list.Add(t); list.Save(); |
Automatic Save
The list will be saved on every change: adding, deleting or updating an item saves the list. You still can call Save and SaveAsync if you want.
|
1 2 3 4 5 |
var t = new Tweet(); var list = CreateList<Tweet>.SavingAt("tweets.data") .UsingJsonSerializer() .SavingWhenCollectionChanges(); list.Add(t); //list saved |
Just note that it can take some time to save, specially on loops (use AddRange when on loops).
Background Save
This is really useful if your list changes a lot, specilly by multithread applications (web). The list will be saved every X seconds, but only if it was modified. You still can call Save and SaveAsync if you want.
|
1 2 3 4 5 6 7 8 9 |
var t = new Tweet(); var list = CreateList<Tweet>.SavingAt("tweets.data") .UsingJsonSerializer() .BackgroundSavingEverySecond(); //or ... var list = CreateList<Tweet>.SavingAt("tweets.data") .UsingJsonSerializer() .BackgroundSavingEvery(TimeSpan.FromSeconds(5)); list.Add(t); //it will be saved on next loop, in a background thread |
Web Applications
Web applications are multithread applications by nature and there will be a lot of requests trying to access our persistent list at the same time. Don’t worry, the list is thread-safe. Just make sure to register it as a singleton on your DI of choice, use the Background Save strategy and you’re all set.
Performance
I did some non-scientific-quick-and-dirty performance tests just to have an idea about how big MiniBiggy can get.
Here is the result:
| Number of Items | Save | Load | Find <3 | File on Disk | Memory |
|---|---|---|---|---|---|
| 1,000 | 0.120s | 0.005s | 0.001s | 61kB | 0.09MB |
| 1,000,000 | 0.350s | 0.335s | 0.005s | 6.54MB | 9.91MB |
| 10,000,000 | 3.248s | 3.985s | 0.039s | 68.345MB | 99.17MB |
As you can see, things are really fast until you reach some millions of registers.
Is mini-biggy for you?
Mini-biggy is an excellent choice for storing your persistence data if:
- you want a freaking fast and simple way to store your objects.
- you have less than a couple million objects (beware for mobile devices)
- you want full Linq support
- your objects are json serialized and you will take care of them when changing your model
Where is the source code? How can I help?
Find more about the project on https://github.com/mini-biggy/mini-biggy!
About
I am 19 years experienced founder and hands-on CTO with deep technical background and community footprint. Passionate about cloud-native applications, automation and increasing team performance.
