mem_dbg-rs

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

(★ 43)

A trait + derive to recursively compute / print the memory usage of a data structure.

mem_dbg-rs Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# mem_dbg

[![crates.io](https://img.shields.io/crates/v/mem_dbg.svg)](https://crates.io/crates/mem_dbg)
[![docs.rs](https://docs.rs/mem_dbg/badge.svg)](https://docs.rs/mem_dbg)
[![rustc](https://img.shields.io/badge/rustc-1.85+-red.svg)](https://rust-lang.github.io/rfcs/2495-min-rust-version.html)
[![CI](https://github.com/zommiommy/mem_dbg-rs/actions/workflows/rust.yml/badge.svg)](https://github.com/zommiommy/mem_dbg-rs/actions)
![license](https://img.shields.io/crates/l/mem_dbg)
[![downloads](https://img.shields.io/crates/d/mem_dbg)](https://crates.io/crates/mem_dbg)
[![coveralls](https://coveralls.io/repos/github/zommiommy/mem_dbg-rs/badge.svg?branch=main)](https://coveralls.io/github/zommiommy/mem_dbg-rs?branch=main)

Traits and associated procedural macros to inspect recursively the memory usage
and layout of a value.

The trait [`MemSize`] can be used to compute the overall memory usage of a value
in bytes; the standard library function [`core::mem::size_of`] returns the
_stack_ size of a type in bytes, but it does not take into consideration heap
memory. We provide implementations for most basic types, a derive macro for
structs and enums whose fields implement [`MemSize`], and support for a few other
crates via optional features.

The trait [`MemDbg`], which depends on [`MemSize`], can be used to display the
recursive layout of a value, together with the size of each part and the
associated padding bytes.

## Why `MemSize`

Other traits partially provide the functionality of [`MemSize`], but either they
require implementing manually a trait, which is prone to error, or they do not
provide the flexibility necessary for [`MemDbg`]. Most importantly, [`MemSize`]
uses the type system to avoid iterating over the content of a container (a
vector, etc.) when it is not necessary, making it possible to compute instantly
the size of values occupying hundreds of gigabytes of heap memory.

This is the result of the benchmark `btree_set_comp` contained in the `examples`
directory. It builds a B-tree set with a hundred million `usize` entries and
then measures its heap size:

```text
Allocated:    3428571500
get_size:     1600000024 349387500 ns
deep_size_of: 1800000024 284149583 ns
mem_size:     3416666554 41 ns
```

The first line is the number of bytes allocated by the program as returned by
[`cap`]. Then, we display the result of [`get-size`], [`deepsize`], and our own
[`MemSize`]. Note that the first two crates are just measuring the space used by
the items, and not by the data structure. Moreover, all other crates are about
seven orders of magnitude slower than our implementation, due to the necessity
to iterate over all elements.

In general, while the size estimation of [`BTreeSet`], [`BTreeMap`], [`HashSet`],
and [`HashMap`] is heuristic in all libraries, `mem_dbg` is significantly more
precise.

The following table compares the [`MemSize`] trait from this crate against the
crates [`deepsize`] and [`get-size`]. The true memory usage (0% error) is
again calculated using the allocator from the [`cap`] crate.

| Type  | Container | Crate          | Error (%)       | Type   | Container | Crate          | Error (%)       |
| ----- | --------- | -------------- | --------------- | ------ | --------- | -------------- | --------------- |
| usize | BTreeMap  | `deep_size_of` | 36.51 ± 20.88   | String | BTreeMap  | `deep_size_of` | 21.53 ± 8.88    |
|       |           | `get_size`     | 42.01 ± 21.73   |        |           | `get_size`     | 16.04 ± 7.74    |
|       |           | `mem_size`     | **1.25 ± 2.59** |        |           | `mem_size`     | **3.17 ± 1.45** |
|       | BTreeSet  | `deep_size_of` | 28.18 ± 12.41   |        | BTreeSet  | `deep_size_of` | 18.58 ± 8.51    |
|       |           | `get_size`     | 46.91 ± 23.70   |        |           | `get_size`     | 17.54 ± 8.38    |
|       |           | `mem_size`     | **1.57 ± 2.97** |        |           | `mem_size`     | **3.73 ± 2.65** |
|       | HashMap   | `deep_size_of` | 15.64 ± 6.34    |        | HashMap   | `deep_size_of` | 6.12 ± 3.42     |
|       |           | `get_size`     | 15.64 ± 6.34    |        |           | `get_size`     | 6.12 ± 3.42     |
|       |           | `mem_size`     | **0.00 ± 0.00** |        |           | `mem_size`     | **0.62 ± 1.30** |
|       | HashSet   | `deep_size_of` | 19.57 ± 7.91    |        | HashSet   | `deep_size_of` | 6.91 ± 3.82     |
|       |           | `get_size`     | 19.57 ± 7.91    |        |           | `get_size`     | 6.91 ± 3.82     |
|       |           | `mem_size`     | **0.00 ± 0.00** |        |           | `mem_size`     | **1.13 ± 2.32** |

## References

Two flags, [`SizeFlags::FOLLOW_REFS`] and [`DbgFlags::FOLLOW_REFS`], make it
possible to follow references when computing the size or displaying the layout
of a value. Analogously, [`SizeFlags::FOLLOW_RCS`] and [`DbgFlags::FOLLOW_RCS`]
make it possible to follow [`Rc`]/[`Arc`] smart pointers.

In both cases, references will be accounted for (when computing size) and
followed (when displaying the layout), only at their first instance. Subsequent
instances will just display an arrow followed by a pointer (see the
[`SizeFlags::FOLLOW_REFS`] documentation for the limitations of our approach).

Because each address is recorded the first time it is seen, following a value
that contains a cycle (for example an `Rc` or reference that, directly or
indirectly, points back to itself) terminates rather than recursing forever:
the allocation behind the cycle is still counted exactly once.

## Padding

The trait [`MemDbg`] is useful to display the layout of a value and understand
how much memory is used by each part. In particular, it exploits the new stable
macro [`std::mem::offset_of`] to display the padding of each field in square
brackets; moreover, the flag [`DbgFlags::RUST_LAYOUT`] makes it possible to
display structures in the layout used by the Rust compiler, rather than
the one given by declaration order.

These features are also available for enums using the feature `offset_of_enum`,
which however needs the nightly compiler, as it enables the unstable feature
`offset_of_enum`.

## Features

- `std`: enables the use of the standard library; this is enabled by default.
- `derive`: enables the derive macros [`MemSize`] and [`MemDbg`]; this is enabled by
  default.
- `offset_of_enum`: support for padding and for the [`DbgFlags::RUST_LAYOUT`] flag
  for enums. Requires the nightly compiler as it enables the unstable feature
  `offset_of_enum`. Calling `mem_dbg` with the flag [`DbgFlags::RUST_LAYOUT`]
  without this feature enabled will result in a panic.
- `half`: support for the [`half`] crate.
- `maligned`: support for the [`maligned`] crate.
- `mmap-rs`: support for the [`mmap-rs`] crate.
- `rand`: support for the [`rand`] crate.
- `aliasable`: support for the [`aliasable`] crate (but [`SizeFlags::CAPACITY`]
  is a no-op for `AliasableVec` and `AliasableString`).
- `maybe-dangling`: support for the [`maybe-dangling`] crate.

## Examples

This is an example program using [`MemSize`] and [`MemDbg`]. Note that we cannot
visualize the effect of the useful [`DbgFlags::COLOR`] flag, which colorizes
sizes depending on their magnitude.

```rust
# #![cfg_attr(feature = "offset_of_enum", feature(offset_of_enum))]
# fn main() -> Result<(), Box<dyn std::error::Error>> {

# #[cfg(all(feature = "std", feature = "derive"))]
# {
use mem_dbg::*;

#[derive(MemSize, MemDbg)]
struct Struct<A, B> {
    a: A,
    b: B,
    test: isize,
}

#[derive(MemSize, MemDbg)]
struct Data<A> {
    a: A,
    b: Vec<i32>,
    c: (u8, String),
}

#[derive(Clone, Copy, MemSize, MemDbg)]
#[mem_size(flat)]
enum TestEnum {
    Unit,
    Unit2(),
    Unit3 {},
    Unnamed(usize, u8),
    Named { first: usize, second: u8 },
}

let b = Vec::with_capacity(100);

let s = Struct {
    a: TestEnum::Unnamed(0, 16),
    b: Data {
        a: vec![0x42_u8; 700],
        b,
        c: (1, "foo".to_owned()),
    },
    test: -0xbadf00d,
};

println!("size:     {}", s.mem_size(SizeFlags::default()));
println!("capacity: {}", s.mem_size(SizeFlags::CAPACITY));
println!();

s.mem_dbg(DbgFlags::empty())?;

println!();

println!("size:     {}", s.mem_size(SizeFlags::default()));
println!("capacity: {}", s.mem_size(SizeFlags::CAPACITY));
println!();

s.mem_dbg(DbgFlags::default() | DbgFlags::CAPACITY | DbgFlags::HUMANIZE)?;

#[cfg(feature = "offset_of_enum")]
{
    println!();

println!("size:     {}", s.mem_size(SizeFlags::default()));
    println!("capacity: {}", s.mem_size(SizeFlags::CAPACITY));
    println!();

s.mem_dbg(DbgFlags::empty() | DbgFlags::RUST_LAYOUT)?;
}
# }
# Ok(())
# }
```

The previous program prints:

```text
size:     807
capacity: 1207

807 B ⏺
 16 B ├╴a
      │ ├╴Variant: Unnamed
  8 B │ ├╴0
  1 B │ ╰╴1
783 B ├╴b
724 B │ ├╴a
 24 B │ ├╴b
 35 B │ ╰╴c
  1 B │   ├╴0 [7B]
 27 B │   ╰╴1
  8 B ╰╴test

size:     807
capacity: 1207

1.207 kB 100.00% ⏺: readme::main::Struct<readme::main::TestEnum, readme::main::Data<alloc::vec::Vec<u8>>>
   16  B   1.33% ├╴a: readme::main::TestEnum
                 │ ├╴Variant: Unnamed
    8  B   0.66% │ ├╴0: usize
    1  B   0.08% │ ╰╴1: u8
1.183 kB  98.01% ├╴b: readme::main::Data<alloc::vec::Vec<u8>>
  724  B  59.98% │ ├╴a: alloc::vec::Vec<u8>
  424  B  35.13% │ ├╴b: alloc::vec::Vec<i32>
   35  B   2.90% │ ╰╴c: (u8, alloc::string::String)
    1  B   0.08% │   ├╴0: u8 [7B]
   27  B   2.24% │   ╰╴1: alloc::string::String
    8  B   0.66% ╰╴test: isize
```

If run with the feature `offset_of_enum`, it prints:

```text
size:     807
capacity: 1207

807 B ⏺
 16 B ├╴a
      │ ├╴Variant: Unnamed
  8 B │ ├╴0
  1 B │ ╰╴1 [6B]
783 B ├╴b
724 B │ ├╴a
 24 B │ ├╴b
 35 B │ ╰╴c
  1 B │   ├╴0 [7B]
 27 B │   ╰╴1
  8 B ╰╴test

size:     807
capacity: 1207

1.207 kB 100.00% ⏺: readme::main::Struct<readme::main::TestEnum, readme::main::Data<alloc::vec::Vec<u8>>>
   16  B   1.33% ├╴a: readme::main::TestEnum
                 │ ├╴Variant: Unnamed
    8  B   0.66% │ ├╴0: usize
    1  B   0.08% │ ╰╴1: u8 [6B]
1.183 kB  98.01% ├╴b: readme::main::Data<alloc::vec::Vec<u8>>
  724  B  59.98% │ ├╴a: alloc::vec::Vec<u8>
  424  B  35.13% │ ├╴b: alloc::vec::Vec<i32>
   35  B   2.90% │ ╰╴c: (u8, alloc::string::String)
    1  B   0.08% │   ├╴0: u8 [7B]
   27  B   2.24% │   ╰╴1: alloc::string::String
    8  B   0.66% ╰╴test: isize

size:     807
capacity: 1207

807 B ⏺
783 B ├╴b
724 B │ ├╴a
 24 B │ ├╴b
 35 B │ ╰╴c
  1 B │   ├╴0 [7B]
 27 B │   ╰╴1
 16 B ├╴a
      │ ├╴Variant: Unnamed
  1 B │ ├╴1 [6B]
  8 B │ ╰╴0
  8 B ╰╴test
```

## Caveats

- We support out-of-the-box most basic types, and tuples up to size ten. The
  derive macros `MemSize`/`MemDbg` will generate implementations for structs and
  enums whose fields implement the associated interface: if this is not the case
  (e.g., because of the orphan rule) one can implement the traits manually.
  A manual `MemSize::mem_size_rec` implementation must always return at least
  `core::mem::size_of_val(self)`: wrapper and derive implementations subtract
  inline field storage before adding recursive contributions.

- [`RefCell`] contents can be followed only if the [`RefCell`] is not mutably
  borrowed; `MemDbg` will show a `<mutably borrowed>` message, but `MemSize`
  will just silently return the size of the `RefCell` itself.

- If you invoke the methods of this crate on a shared reference, the compiler
  will automatically dereference it, and the method will be invoked on the
  referenced type:

```rust
# fn main() -> Result<(), Box<dyn std::error::Error>> {
use mem_dbg::*;

let mut x: [i32; 4] = [0, 0, 0, 0];

assert_eq!(
    (&x).mem_size(SizeFlags::default()),
    std::mem::size_of::<[i32; 4]>()
);

assert_eq!(
    (&mut x).mem_size(SizeFlags::default()),
    std::mem::size_of::<&mut [i32; 4]>()
);

assert_eq!(
    <&[i32; 4] as MemSize>::mem_size(&&x, SizeFlags::default()),
    std::mem::size_of::<&[i32; 4]>()
);
# Ok(())
# }
```

- Computation of the size of arrays, slices, vectors, or container types, will
  be performed by iterating over their elements unless the type is flat.
  See [`FlatType`] for more details.

- When all fields of a struct or enum implement `FlatType<Flat=True>`, a
  compile-time error will suggest adding `#[mem_size(flat)]` (if the type is
  flat) or `#[mem_size(rec)]` (to explicitly opt out of the
  optimization and silence the error).

- The content of vectors and slices is not expanded recursively as the output
  might be too complex; this might change in the future (e.g., via a flag)
  should interesting use cases arise.

- Unions are not supported. See the section below for a manually written
  example.

## Unions

Unions have no discriminant tag, so the library cannot know which field is
active. The recommended solution is to create `#[repr(transparent)]` wrappers,
one per variant, each encoding which field is active at the type level, and
delegate the implementation of the traits to the active field. For example:

```rust
# fn main() -> Result<(), Box<dyn std::error::Error>> {
# #[cfg(feature = "std")]
# {
use mem_dbg::*;

union IntOrFloat {
    i: i32,
    f: f32,
}

/// Wrapper that tells the library the `i` field is active.
#[repr(transparent)]
struct IntOrFloatI(IntOrFloat);

/// Wrapper that tells the library the `f` field is active.
#[repr(transparent)]
struct IntOrFloatF(IntOrFloat);

impl FlatType for IntOrFloatI {
    type Flat = True;
}

impl MemSize for IntOrFloatI {
    fn mem_size_rec(
        &self,
        _flags: SizeFlags,
        _refs: &mut HashMap<usize, usize>,
    ) -> usize {
        core::mem::size_of::<Self>()
    }
}

impl MemDbgImpl for IntOrFloatI {
    fn _mem_dbg_rec_on(
        &self,
        writer: &mut impl core::fmt::Write,
        total_size: usize,
        max_depth: usize,
        prefix: &mut String,
        _is_last: bool,
        flags: DbgFlags,
        dbg_refs: &mut HashSet<usize>,
    ) -> core::fmt::Result {
        unsafe { self.0.i }._mem_dbg_depth_on(
            writer,
            total_size,
            max_depth,
            prefix,
            Some("i"),
            true,
            core::mem::size_of::<Self>(),
            flags,
            dbg_refs,
        )
    }
}

impl FlatType for IntOrFloatF {
    type Flat = True;
}

impl MemSize for IntOrFloatF {
    fn mem_size_rec(
        &self,
        _flags: SizeFlags,
        _refs: &mut HashMap<usize, usize>,
    ) -> usize {
        core::mem::size_of::<Self>()
    }
}

impl MemDbgImpl for IntOrFloatF {
    fn _mem_dbg_rec_on(
        &self,
        writer: &mut impl core::fmt::Write,
        total_size: usize,
        max_depth: usize,
        prefix: &mut String,
        _is_last: bool,
        flags: DbgFlags,
        dbg_refs: &mut HashSet<usize>,
    ) -> core::fmt::Result {
        unsafe { self.0.f }._mem_dbg_depth_on(
            writer,
            total_size,
            max_depth,
            prefix,
            Some("f"),
            true,
            core::mem::size_of::<Self>(),
            flags,
            dbg_refs,
        )
    }
}

let w = IntOrFloatI(IntOrFloat { i: 42 });
assert_eq!(w.mem_size(SizeFlags::default()), 4);
w.mem_dbg(DbgFlags::empty())?;
# }
# Ok(())
# }
```

[`MemDbg`]: https://docs.rs/mem_dbg/latest/mem_dbg/trait.MemDbg.html
[`MemSize`]: https://docs.rs/mem_dbg/latest/mem_dbg/trait.MemSize.html
[`core::mem::size_of`]: https://doc.rust-lang.org/core/mem/fn.size_of.html
[`DbgFlags::RUST_LAYOUT`]: https://docs.rs/mem_dbg/latest/mem_dbg/struct.DbgFlags.html#associatedconstant.RUST_LAYOUT
[`DbgFlags::COLOR`]: https://docs.rs/mem_dbg/latest/mem_dbg/struct.DbgFlags.html#associatedconstant.COLOR
[`FlatType`]: https://docs.rs/mem_dbg/latest/mem_dbg/trait.FlatType.html
[`cap`]: https://crates.io/crates/cap
[`get-size`]: https://crates.io/crates/get_size
[`deepsize`]: https://crates.io/crates/deepsize
[`maligned`]: https://crates.io/crates/maligned
[`mmap-rs`]: https://crates.io/crates/mmap-rs
[`half`]: https://crates.io/crates/half
[`rand`]: https://crates.io/crates/rand
[`aliasable`]: https://crates.io/crates/aliasable
[`maybe-dangling`]: https://crates.io/crates/maybe-dangling
[`Rc`]: https://doc.rust-lang.org/std/rc/struct.Rc.html
[`Arc`]: https://doc.rust-lang.org/std/sync/struct.Arc.html
[`DbgFlags::FOLLOW_REFS`]: https://docs.rs/mem_dbg/latest/mem_dbg/struct.DbgFlags.html#associatedconstant.FOLLOW_REFS
[`DbgFlags::FOLLOW_RCS`]: https://docs.rs/mem_dbg/latest/mem_dbg/struct.DbgFlags.html#associatedconstant.FOLLOW_RCS
[`SizeFlags::FOLLOW_REFS`]: https://docs.rs/mem_dbg/latest/mem_dbg/struct.SizeFlags.html#associatedconstant.FOLLOW_REFS
[`SizeFlags::FOLLOW_RCS`]: https://docs.rs/mem_dbg/latest/mem_dbg/struct.SizeFlags.html#associatedconstant.FOLLOW_RCS
[`SizeFlags::CAPACITY`]: https://docs.rs/mem_dbg/latest/mem_dbg/struct.SizeFlags.html#associatedconstant.CAPACITY
[`HashSet`]: https://doc.rust-lang.org/std/collections/struct.HashSet.html
[`HashMap`]: https://doc.rust-lang.org/std/collections/struct.HashMap.html
[`BTreeSet`]: https://doc.rust-lang.org/std/collections/struct.BTreeSet.html
[`BTreeMap`]: https://doc.rust-lang.org/std/collections/struct.BTreeMap.html
[`RefCell`]: https://doc.rust-lang.org/std/cell/struct.RefCell.html
[`Copy`]: https://doc.rust-lang.org/std/marker/trait.Copy.html
[`core::mem::offset_of`]: https://doc.rust-lang.org/core/mem/macro.offset_of.html

mem_dbg-rs

mem_dbg-rs Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

mem_dbg-rs

mem_dbg-rs Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README