Using Vuex getters as an efficient ORM

vuex
performance

#1

Summary

I have built a large, very data-heavy Vue application, and as it grew it started becoming sluggish and inefficient with memory. This was due in large part to the way that I was accessing data from my Vuex store modules. I have learned some great lessons that have helped to dramatically improve the efficiency of my app, and I would like to share them, specifically the changes that I made to my Vuex stores.

If you’d like to skip ahead and just look at an implementation, I created a proof of concept for my Vuex solution here: https://github.com/aidangarza/vuex-indexed-getters

Vuex

I have a central Vuex store for my whole application, and a Vuex module for each resource that I work with (location, business, report, etc).

Each resource’s store module had a state property called all where I deposited all instances of that resource returned from a list request to a web-service.

I’ll talk about how I used to organize each store module, and then how I changed it to make it way more efficient.

The OLD way (bad)

The state.all list works fine as it is when I need to display the full list in a component; however, most components are only concerned with a single instance of a resource. To handle that, I made Vuex getter functions that acted like object-relational mapping (ORM) tools; for example, converting the the state.all array into an object where each property was an item’s id, and the value was the item.

Old Store Module Example

    export default {
      state: () => ({
        all: [
          { id: 1, tags: ['tag1', 'tag2'] },
          ...
        ],
        ...
      }),

      ...

      getters: {
        byId: (state) => {
          return state.all.reduce((map, item) => {
            map[item.id] = item
            return map
          }, {})
        },

        byTag: (state) => {
          return state.all.reduce((map, item, index) => {
            for (let i = 0; i < item.tags.length; i++) {
              map[item.tags[i]] = map[item.tags[i]] || []
              map[item.tags[i]].push(item)
            }
            return map
          }, {})
        },
      }
    }

Old Component Example

    export default {
      ...,

      data () {
        return {
          itemId: 1
        }
      },

      computed: {
        item () {
          return this.$store.getters['path/to/byId'][this.itemId]
        },

        relatedItems () {
          return this.item && this.item.tags.length
            ? this.$store.getters['path/to/byTag'][this.item.tags[0]]
            : []
        }
      }
    }

For small lists of small objects, this works fine; however, as the resources get bigger, and the user creates more of them, this method quickly results in a lot of duplicated data stored in memory. As some users generated hundreds or even thousands of resource instances, it became nearly unusable.

The NEW way (good!)

To fix this problem, I looked to an old, standard practice in programming: indexing. Instead of storing a map with the full item values duplicated in the getter, you can store a map to the index of the item in state.all. Then, you can create a new getter that returns a function to access a single item. In my experience, the indexing getter functions always run faster than the old getter functions, and their output takes up a lot less space in memory (on average 80% less in my app).

New Store Module Example

    export default {
      state: () => ({
        all: [
          { id: 1, tags: ['tag1', 'tag2'] },
          ...
        ],
        ...
      }),

      ...

      getters: {
        indexById: (state) => {
          return state.all.reduce((map, item, index) => {
            // Store the `index` instead of the `item`
            map[item.id] = index
            return map
          }, {})
        },

        byId: (state, getters) => (id) => {
          return state.all[getters.indexById[id]]
        },

        indexByTags: (state) => {
          return state.all.reduce((map, item, index) => {
            for (let i = 0; i < item.tags.length; i++) {
              map[item.tags[i]] = map[item.tags[i]] || []
              // Again, store the `index` not the `item`
              map[item.tags[i]].push(index)
            }
            return map
          }, {})
        },

        byTag: (state, getters) => (tag) => {
          return (getters.indexByTags[tag] || []).map(index => state.all[index])
        }
      }
    }

New Component Example

    export default {
      ...,

      data () {
        return {
          itemId: 1
        }
      },

      computed: {
        item () {
          return this.$store.getters['path/to/byId'](this.itemId)
        },

        relatedItems () {
          return this.item && this.item.tags.length
            ? this.$store.getters['path/to/byTag'](this.item.tags[0])
            : []
        }
      }
    }

The change seems small, but it makes a huge difference in terms of performance and memory efficiency. It is still fully reactive, just as before, but we’re no longer duplicating all of the resource objects in memory. In my implementation, I abstracted out the various indexing methodologies and index expansion methodologies to make the code very maintainable.

You can check out a full proof of concept on github, here: https://github.com/aidangarza/vuex-indexed-getters

Let me know if you have any questions, or if you have found any ways to make dealing with large data sets in a Vuex store even more efficient.


#2

Thanks for sharing you experience. I also had a similar problem but solved it on a different way.

First of all, for getters that are called with a parameter, I created a Cache module that caches array.filter results, in order to reduce array traversing.

Secondly, I offloaded array traversing to a WebWorker in order not to block the main thread. Btw, I recommend a great tool: workerize

Because of this change I had to convert my getters into actions, because only actions can return a Promise.

So, in your code you could also use a WebWorker to create the indexes, especially if you work with huge collections. I like the indexes idea, imo it’s better than the caching technique that I use. But it’s good only if getters take just one argument.


#3

Funnily enough, soon after I posted this, I started thinking about web workers for building the indexes. I’m using Simple Web Worker, but Workerize looks excellent for use with Webpack. Thanks for the tip!

My plan is to use the Vuex “subscribe to mutation” feature to call an indexing action every time I modify the list in the state, and then just store the index objects in the state.