Python's Difflib SequenceMatcher Speed Up

I'm using difflib SequenceMatcher (ratio() method) to define similarity between text files. While difflib is relatively fast to compare a small set of text files e.g. 10 files of 7

Solution 1:

The issue you're finding is very common, since difflib is not optimized. Here are some tricks I've found over the years while developing a tool that compares HTML documents.

Files fit in memory

Create two lists, containing the lines from each file. Then call difflib.SequenceMatcher with the lists as parameters. The SequenceMatcher knows how to handle lists, and the process will be much faster since it is done on a line by line basis instead of char by char. This might reduce the precision.

Take a look at fuzzy_string_cmp.py and diff.py to see how I'm doing exactly this.

Alternative

There is a great library called diff_match_patch which is available in pypi. The library will perform fast diffs between two strings and return the changes (line added, line equal, line removed).

By leveraging diff_match_patch you should be able to create your own dmp_quick_ratio function.

In diff.py you can see how I'm using the library to get inspiration for creating dmp_quick_ratio.

My tests showed that using diff_match_patch was 20 times faster than Python's difflib.

Solution 2:

There is a C implementation of difflib.SequenceMatcher, cdifflib.

Replace the SequenceMatcher and all difflib operations will be about 4x faster

from cdifflib import CSequenceMatcher
import difflib
difflib.SequenceMatcher = CSequenceMatcher

Solution 3:

You can get a small speedup using pypy

http://pypy.org/