also discuss a real world use case where TinyBits is used to encode and transmit instrumentation data from a Ruby app. And I compare it to JSON and Msgpack
Beating them both by varying degrees in all tested metrics, payload size, process memory & performance overhead
github.com/oldmoe/tinyb...
I explain more about the new features and its benefits on my blog
t.co/FOJszyddCn
TinyBits 0.6.0 is released, better memory management, smaller multi-object buffers and a more robust dictionary support
With external dictionaries, a schema-less encoder like TinyBits can match (and sometimes exceed) the compactness of Protocol Buffers encoded output
New gems and updates from @Mohammad Ali @Sergio Bayona @Joshua Young @Étienne Barrié @Simon J @Brad Gessler @Stan Lo @Bozhidar Batsov @Avo and more
New gems and updates from @oldmoe.bsky.social @Sergio Bayona @joshuay03.bsky.social @bihi.bsky.social @simonjuk.bsky.social @bradgessler.com @st0012.dev @batsov.net @avohq.io and more
This also opens the door for some creative transformations, you can for example have a dict with enum values [glücklich, traurig, gleichgültig] on one side and [happy, sad, indifferent] on the other
Yes of course, you need to have the exact same dictionaries (in the exact same order) on both sides of the wire. Similar to a dictionary used with zstd or lz4. Keeping those in sync is on the user
Thanks for the catch, I pasted the wrong data, fixed now
This is basically protocol buffers (and other schema based encoders) territory, but while being 10X simple
Read more here:
GitHub.com/oldmoe/tinybits-rb/ADVANCED.md
TinyBits is getting a new interesting feature, you can now (using the repo's head) supply the pack/unpack method with an external dictionary.
For shorter messages with no duplicate strings this can lead to a dramatic size reduction, from 96 to 34 bytes in the example below
I built this to communicate data between services, much like you would use JSON or MessagePack, just faster and smaller, specifically for the types of payloads I deal with
TinyBits delivering better performance on all metrics
For a data set of 11 different documents, the TinyBits Ruby gem was faster that all other schemaless encoders tested (Oj, JSON, Msgpack & CBOR) for encoding and decoding, all while producing much smaller serialized sizes.
Unlike other binary formats, it was as compressible as text formats like JSON
TinyBits: A new schema-less binary serialization format. Fast to encode, fast to decode and generally smaller output than all other schema-less formats.
Currently available as a preview for C and Ruby. A Python module is also in the works.
oldmoe.blog/2025/05/05/s...
I will share all the data samples soon in a repo. I believe the issue is rendering numbers as text, specially doubles with larger decimal values. Rendering these will always be more expensive than just writing the 8 bytes that make up the 64bit double for binary foramts
If you are sending schema-less data from Ruby over the wire, or if you are storing it to disk, what is your best option among the current offering?
I went through the most prominent serializers and got some interesting results
oldmoe.blog/2025/04/21/r...
