[Ndn-interest] Announcing "persist", a fast and easy-to-use NDN persistent content store

[Tai-Lin Chu]

# Write 8.820665ms (8.82µs/op) (113378 op/sec)
# Read 1.000094901s (49ns/op) (20408163 op/sec)

Yes, but this is probably meaningless.

> maximum amount of data you can store without significant drop of read/write speed

https://twitter.com/boltdb/status/538129302702010368

> average response time for read/write operations

I think ACID is still relevant for persistent ndn content store;
otherwise we will have some even faster options (other than boltdb).
Boltdb is (very) good at read and bad at (random) write, but its RW is
still faster than sqlite by far.

> memory scalability (does memory consumption go up linearly or sub-linearly with the amount of stored data?)

mmap just throws cache management back to the OS. The memory will
increase steadily but the OS can reclaim it later.


> I have been troubled by various issues with repo-ng for a very long time. It would be great if we can have a more performant alternative. I'd be interested in contributing by adding the repo command protocols on top of it.

great!

On Thu, Apr 30, 2015 at 4:13 PM, Wentao Shang <wentaoshang at gmail.com> wrote:
> I see. So I see this as a potential replacement for repo-ng. Wonder if you
> have done any benchmark on it? The things I care about include: maximum
> amount of data you can store without significant drop of read/write speed;
> average response time for read/write operations; memory scalability (does
> memory consumption go up linearly or sub-linearly with the amount of stored
> data?); disk space requirement.
>
> I have been troubled by various issues with repo-ng for a very long time. It
> would be great if we can have a more performant alternative. I'd be
> interested in contributing by adding the repo command protocols on top of
> it.
>
> Thanks,
> Wentao
>
> On Thu, Apr 30, 2015 at 3:59 PM Tai-Lin Chu <tailinchu at gmail.com> wrote:
>>
>> It uses memory-mapped files, so it has the read performance of a pure
>> in-memory storage while still offering the persistence of standard
>> disk-based storage.
>>
>> > a replacement for ndn repo?
>>
>> No, it does not have management protocol now, but it is very easy to
>> create another ndn repo with "persist".
>>
>> > a new implementation of content store (i.e. cache) for ndn routers?
>>
>> No, It is for any ndn app. If a router needs disk-based storage, it
>> can also use "persist".
>>
>> > or a new implementation of in-memory cache that can be used by ndn
>> > applications?
>>
>> No, the term "persist" in ndn-cxx in-memory cache means
>> "eviction-less". This is entirely different.
>>
>> Thanks.
>>
>>
>>
>>
>> On Thu, Apr 30, 2015 at 3:34 PM, Wentao Shang <wentaoshang at gmail.com>
>> wrote:
>> > The phrase "persistent content store" confuses me. Could you clarify
>> > that
>> > this is intended as a replacement for ndn repo, or a new implementation
>> > of
>> > content store (i.e. cache) for ndn routers, or a new implementation of
>> > in-memory cache that can be used by ndn applications?
>> >
>> > Thanks,
>> > Wentao
>> >
>> > On Thu, Apr 30, 2015 at 3:09 PM Tai-Lin Chu <tailinchu at gmail.com> wrote:
>> >>
>> >> "persist" uses BoltDB, which is based on LMDB. LMDM's super clever
>> >> design made the key/value store very fast for reads and safe for
>> >> writes. MVCC-like design with copy-on-write B+trees avoids locking
>> >> database for reads during writes, and makes it possible to provide
>> >> full ACID transactions support.
>> >>
>> >> I believe this is the first attempt to create a high performance and
>> >> persistent content store for NDN project. Comments are welcomed :)
>> >>
>> >> Project page:
>> >> https://github.com/go-ndn/persist
>> >>
>> >> Example:
>> >> https://github.com/go-ndn/example/blob/master/producer/producer.go#L44
>> >>
>> >> Cheers!
>> >> _______________________________________________
>> >> Ndn-interest mailing list
>> >> Ndn-interest at lists.cs.ucla.edu
>> >> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest