Dealing with mega data in angular

A solution to handling large datasets in angular

A while back I reflected on performance issues when using angular and proposed solutions in order to overcome them.

One of the main performance issues that arises when using angular is the overuse of ng-repeat. An issue that is particularly noticeable when dealing with large datasets that are being rendered in a tabular style.

Some might opt for the simplest solution of using ng-repeat. An example might be

<table>
  <tr ng-repeat="item in items">
    <td>{{item.name}}</td>
  </tr>
</table>

which could be fine with a small amount of data. But, if we start dealing with datasets that consist of hundreds of items, we are going to hit issues. The problem arises because of angulars dirty checking solution. In layman’s terms, with each new table row, angular is gaining more things to be watched and observed. The more things that need to be checked and observed the slower our application will become until it becomes unusable.

So what’s the solution then?

The solution is to make use of directives. Using directives we can interact directly with the DOM and make use of quick DOM manipulations whilst also ridding angular of the burden of unnecessary watches.

For those in camp TL;DR — I have put together a directive “ng-mega-table”, which can be grabbed from here. In the repo you’ll find code and a runnable demo so you can run with datasets consisting of 1000s of items.

ng-mega-table: The directive solution

What makes for a simple solution is to make a directive that allows us to interact with the DOM directly. The biggest hurdle with this approach is being able to generate a large amount of dynamic markup given a dataset that will generating both the head and body content of our table. The solution is to use a templating framework such as Handlebars along with javascript techniques such as event delegation.

The best thing to do is to take a look at some of the code in the repo. But, below are some more details of the main features that make the directive approach possible.

Templating with Handlebars.js

Using a minimal templating library such as Handlebars.js allows us to rapidly template incoming data against predefined templates and produce markup for our tables content.

Here is the function for rendering the table body

renderTBody = function(data) {
  var $tbody, $tbodyTemplate;
  $tbody = $elem.find(‘tbody’);
  $tbodyTemplate = Handlebars.compile($templateCache.get(‘ngMegaTableBody.html’));
  $scope.$apply(function() {
    if (data.length === 0) {
      return $tbody.html($(‘<h4>No data currently available</h4>’));
    } else {
      opts.items = data;
      return $tbody.replaceWith($($tbodyTemplate(opts)));
    }
  });
  return $scope.loading = false;
};

Data is passed in and the resulting markup that is created replaces the current table body.

You may also notice that instead of storing our Handlebars templates within HTML files, we can actually make use of angulars $templateCache and store them craftily within our Javascript meaning no requests having to be made. This makes it quite nice and clean when packaging our directive into a module.

When to update table content

So, how do we know when to update our table content? Instead of watching for data changes. It’s much better for performance if we trigger our update based on an event being fired.

if (opts.changeEvent === undefined) {
  throw new Error(‘ngMegaTable: no data change event defined’);
} else {
  $scope.$on(opts.changeEvent, function(e, data) {
    return renderTBody(data);
  });
}

Here we define that our directive will listen for a change event being fired and on the event being fired, pass event data into our previously mentioned table body rendering function.

This event would be fired from wherever we are making changes to data. In this example, I make use of $rootScope.$broadcast. The following code is from a demo controller.

$scope.data = DataSrv.get(size);
$rootScope.$broadcast(‘data:changed’, $scope.data);

Event handling with delegation

Let’s say we want to add actions to the rows in our table. Adding event listeners to each individual row could be costly. Instead, we can actually make use of event delegation. Instead of adding event listeners to each row of our table, we have one simple event listener for our table that on click checks the click event target.

For the example directive I have gone for a generic solution. Therefore I am actually iterating over the directives table options to determine what event targets should trigger which events.

$elem.on(‘click’, function(e) {
  return [].forEach.call(opts.columns, function(column, index) {
    if (column.type && column.type === ‘action’) {
      if (e && e.target && e.target && (e.target.hasAttribute(column.selector) || e.target.className.indexOf(column.selector) !== -1)) {
        return opts.actions[column.action](e.target.getAttribute(column.actionParamsAttribute));
      }
    }
  });
});

Making a generic solution

A generic solution isn’t the clearest and you’d most likely be best tweaking the directive to your own needs with statically defined templates. However, it is possible and could be fine with smaller examples.

Final thoughts

Dealing with large data sets in angular isn’t taxing but can be problematic in terms of performance. The use of directives provides an effective solution for dealing with large data sets. I ran the demo with datasets in excess of 10,000 items and it rendered without issue.

Initially, I required directives to combat performance in older browsers such as IE8. But I would recommend this approach regardless of the browser or device being viewed on as it will speed up your application. In experience, I’ve seen noticeable improvements on devices too such as the iPhone 4.

As always, any questions or suggestions, please feel free to leave a response or tweet me 🐦!