What Is A Fast Pythonic Way To Deepcopy Just Data From A Python Dict Or List ?

When we need to copy full data from a dictionary containing primitive data types ( for simplicity, lets ignore presence of datatypes like datetime etc), the most obvious choice tha

Solution 1:

It really depends on your needs. deepcopy was built with the intention to do the (most) correct thing. It keeps shared references, it doesn't recurse into infinite recursive structures and so on... It can do that by keeping a memo dictionary in which all encountered "things" are inserted by reference. That's what makes it quite slow for pure-data copies. However I would almost always say that deepcopy is the most pythonic way to copy data even if other approaches could be faster.

If you have pure-data and a limited amount of types inside it you could build your own deepcopy (build roughly after the implementation of deepcopy in CPython):

_dispatcher = {}

def_copy_list(l, dispatch):
    ret = l.copy()
    for idx, item inenumerate(ret):
        cp = dispatch.get(type(item))
        if cp isnotNone:
            ret[idx] = cp(item, dispatch)
    return ret

def_copy_dict(d, dispatch):
    ret = d.copy()
    for key, value in ret.items():
        cp = dispatch.get(type(value))
        if cp isnotNone:
            ret[key] = cp(value, dispatch)

    return ret

_dispatcher[list] = _copy_list
_dispatcher[dict] = _copy_dict

defdeepcopy(sth):
    cp = _dispatcher.get(type(sth))
    if cp isNone:
        return sth
    else:
        return cp(sth, _dispatcher)

This only works correct for all immutable non-container types and list and dict instances. You could add more dispatchers if you need them.

# Timings done on Python 3.5.3 - Windows - on a really slow laptop :-/

import copy
import msgpack
import json

import string

data = {'name':'John Doe','ranks':{'sports':13,'edu':34,'arts':45},'grade':5}

%timeit deepcopy(data)
# 11.9 µs ± 280 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
%timeit copy.deepcopy(data)
# 64.3 µs ± 1.15 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
%timeit json.loads(json.dumps(data))
# 65.9 µs ± 2.53 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
%timeit msgpack.unpackb(msgpack.packb(data))
# 56.5 µs ± 2.53 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Let's also see how it performs when copying a big dictionary containing strings and integers:

data = {''.join([a,b,c]): 1for a in string.ascii_letters for b in string.ascii_letters for c in string.ascii_letters}

%timeit deepcopy(data)
# 194 ms ± 5.37 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
%timeit copy.deepcopy(data)
# 1.02 s ± 46.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit json.loads(json.dumps(data))
# 398 ms ± 20.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit msgpack.unpackb(msgpack.packb(data))
# 238 ms ± 8.81 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Solution 2:

I think you can manually implement what you need by overriding object.__deepcopy__.

A pythonic way to do this is creating your custom dict extends from builtin dict and implement your custom __deepcopy__.

Solution 3:

@MSeifert The suggested answer is not accurate

So far i found ujson.loads(ujson.dumps(my_dict)) to be the fastest option which looks strange (how translating dict to string and then from string to new dict is faster then some pure copy)

Here is an example of the methods i tried and their running time for small dictionary (the results of course are more clear with larger dictionary):

x = {'a':1,'b':2,'c':3,'d':4, 'e':{'a':1,'b':2}}

#this function only handle dict of dicts very similar to the suggested solutiondeffast_copy(d):
    output = d.copy()
    for key, value in output.items():
        output[key] = fast_copy(value) ifisinstance(value, dict) else value        
    return output



from copy import deepcopy
import ujson


%timeit deepcopy(x)
13.5 µs ± 146 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%timeit fast_copy(x)
2.57 µs ± 31.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%timeit ujson.loads(ujson.dumps(x))
1.67 µs ± 14.8 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

is there any other C extension that might work better than ujson? it very strange that this is the fastest method to copy large dict.

Solution 4:

It's always fastest to write your own copy function specific to your data structure.

Your example

data = {
    'name': 'John Doe',
    'ranks': {
        'sports': 13,
        'edu': 34,
        'arts': 45
        },
    'grade': 5
    }

is a dict consisting just of strs or dicts. Hence:

def copy(obj):

    out = obj.copy() # Shallow copyfor k, v in obj.items():

        ifisinstance(obj[k], dict):

            out[k] = obj[k].copy()

    return obj

%timeit deepcopy(data)
5.26 µs ± 88.3 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
%timeit json.loads(json.dumps(data))
5.11 µs ± 117 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
%timeit msgpack.unpackb(msgpack.packb(data))
2.44 µs ± 76.1 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
%timeit ujson.loads(ujson.dumps(data))
1.63 µs ± 25.2 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

%timeit copy(data)
548 ns ± 5.77 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)