How to convert JSON data into a Python object

How to convert JSON data into a Python object

I want to use Python to convert JSON data into a Python object.
I receive JSON data objects from the Facebook API, which I want to store in my database.  
My current View in Django (Python) (request.POST contains the JSON):
response = request.POST
user = FbApiUser(user_id = response['id'])
user.name = response['name']
user.username = response['username']
user.save()


This works fine, but how do I handle complex JSON data objects?   
Wouldn't it be much better if I could somehow convert this JSON object into a Python object for easy use?

Solutions/Answers:

Answer 1:

You can do it in one line, using namedtuple and object_hook:

import json
from collections import namedtuple

data = '{"name": "John Smith", "hometown": {"name": "New York", "id": 123}}'

# Parse JSON into an object with attributes corresponding to dict keys.
x = json.loads(data, object_hook=lambda d: namedtuple('X', d.keys())(*d.values()))
print x.name, x.hometown.name, x.hometown.id

or, to reuse this easily:

def _json_object_hook(d): return namedtuple('X', d.keys())(*d.values())
def json2obj(data): return json.loads(data, object_hook=_json_object_hook)

x = json2obj(data)

If you want it to handle keys that aren’t good attribute names, check out namedtuple‘s rename parameter.

Answer 2:

Check out the section titled Specializing JSON object decoding in the json module documentation. You can use that to decode a JSON object into a specific Python type.

Here’s an example:

class User(object):
    def __init__(self, name, username):
        self.name = name
        self.username = username

import json
def object_decoder(obj):
    if '__type__' in obj and obj['__type__'] == 'User':
        return User(obj['name'], obj['username'])
    return obj

json.loads('{"__type__": "User", "name": "John Smith", "username": "jsmith"}',
           object_hook=object_decoder)

print type(User)  # -> <type 'type'>

Update

If you want to access data in a dictionary via the json module do this:

user = json.loads('{"__type__": "User", "name": "John Smith", "username": "jsmith"}')
print user['name']
print user['username']

Just like a regular dictionary.

Answer 3:

This is not code golf, but here is my shortest trick, using types.SimpleNamespace as the container for JSON objects.

Compared to the leading namedtuple solution, it is:

  • probably faster/smaller as it does not create a class for each object
  • shorter
  • no rename option, and probably the same limitation on keys that are not valid identifiers (uses setattr under the covers)

Example:

from __future__ import print_function
import json

try:
    from types import SimpleNamespace as Namespace
except ImportError:
    # Python 2.x fallback
    from argparse import Namespace

data = '{"name": "John Smith", "hometown": {"name": "New York", "id": 123}}'

x = json.loads(data, object_hook=lambda d: Namespace(**d))

print (x.name, x.hometown.name, x.hometown.id)

Answer 4:

You could try this:

class User(object):
    def __init__(self, name, username, *args, **kwargs):
        self.name = name
        self.username = username

import json
j = json.loads(your_json)
u = User(**j)

Just create a new Object, and pass the parameters as a map.

Answer 5:

Here’s a quick and dirty json pickle alternative

import json

class User:
    def __init__(self, name, username):
        self.name = name
        self.username = username

    def to_json(self):
        return json.dumps(self.__dict__)

    @classmethod
    def from_json(cls, json_str):
        json_dict = json.loads(json_str)
        return cls(**json_dict)

# example usage
User("tbrown", "Tom Brown").to_json()
User.from_json(User("tbrown", "Tom Brown").to_json()).to_json()

Answer 6:

For complex objects, you can use JSON Pickle

Python library for serializing any arbitrary object graph into JSON.
It can take almost any Python object and turn the object into JSON.
Additionally, it can reconstitute the object back into Python.

Answer 7:

If you’re using Python 3.5+, you can use jsons to serialize and deserialize to plain old Python objects:

import jsons

response = request.POST

# You'll need your class attributes to match your dict keys, so in your case do:
response['id'] = response.pop('user_id')

# Then you can load that dict into your class:
user = jsons.load(response, FbApiUser)

user.save()

You could also make FbApiUser inherit from jsons.JsonSerializable for more elegance:

user = FbApiUser.from_json(response)

These examples will work if your class consists of Python default types, like strings, integers, lists, datetimes, etc. The jsons lib will require type hints for custom types though.

Answer 8:

I have written a small (de)serialization framework called any2any that helps doing complex transformations between two Python types.

In your case, I guess you want to transform from a dictionary (obtained with json.loads) to an complex object response.education ; response.name, with a nested structure response.education.id, etc …
So that’s exactly what this framework is made for. The documentation is not great yet, but by using any2any.simple.MappingToObject, you should be able to do that very easily. Please ask if you need help.

Answer 9:

Since noone provided an answer quite like mine, I am going to post it here.

It is a robust class that can easily convert back and forth between json str and dict that I have copied from my answer to another question:

import json

class PyJSON(object):
    def __init__(self, d):
        if type(d) is str:
            d = json.loads(d)

        self.from_dict(d)

    def from_dict(self, d):
        self.__dict__ = {}
        for key, value in d.items():
            if type(value) is dict:
                value = PyJSON(value)
            self.__dict__[key] = value

    def to_dict(self):
        d = {}
        for key, value in self.__dict__.items():
            if type(value) is PyJSON:
                value = value.to_dict()
            d[key] = value
        return d

    def __repr__(self):
        return str(self.to_dict())

    def __setitem__(self, key, value):
        self.__dict__[key] = value

    def __getitem__(self, key):
        return self.__dict__[key]

json_str = """... json string ..."""

py_json = PyJSON(json_str)

Answer 10:

If you are using python 3.6+, you can use marshmallow-dataclass. Contrarily to all the solutions listed above, it is both simple, and type safe:

from marshmallow_dataclass import dataclass

@dataclass
class User:
    name: str

user, err = User.Schema().load({"name": "Ramirez"})

Answer 11:

Improving the lovasoa’s very good answer.

If you are using python 3.6+, you can use:
pip install marshmallow-enum and
pip install marshmallow-dataclass

Its simple and type safe.

You can transform your class in a string-json and vice-versa:

From Object to String Json:

    from marshmallow_dataclass import dataclass
    user = User("Danilo","50","RedBull",15,OrderStatus.CREATED)
    user_json = User.Schema().dumps(user)
    user_json_str = user_json.data

From String Json to Object:

    json_str = '{"name":"Danilo", "orderId":"50", "productName":"RedBull", "quantity":15, "status":"Created"}'
    user, err = User.Schema().loads(json_str)
    print(user,flush=True)

Class definitions:

class OrderStatus(Enum):
    CREATED = 'Created'
    PENDING = 'Pending'
    CONFIRMED = 'Confirmed'
    FAILED = 'Failed'

@dataclass
class User:
    def __init__(self, name, orderId, productName, quantity, status):
        self.name = name
        self.orderId = orderId
        self.productName = productName
        self.quantity = quantity
        self.status = status

    name: str
    orderId: str
    productName: str
    quantity: int
    status: OrderStatus

Answer 12:

Modifying @DS response a bit, to load from a file:

def _json_object_hook(d): return namedtuple('X', d.keys())(*d.values())
def load_data(file_name):
  with open(file_name, 'r') as file_data:
    return file_data.read().replace('\n', '')
def json2obj(file_name): return json.loads(load_data(file_name), object_hook=_json_object_hook)

One thing: this cannot load items with numbers ahead. Like this:

{
  "1_first_item": {
    "A": "1",
    "B": "2"
  }
}

Because “1_first_item” is not a valid python field name.

Answer 13:

While searching for a solution, I’ve stumbled upon this blog post: https://blog.mosthege.net/2016/11/12/json-deserialization-of-nested-objects/

It uses the same technique as stated in previous answers but with a usage of decorators.
Another thing I found useful is the fact that it returns a typed object at the end of deserialisation

class JsonConvert(object):
    class_mappings = {}

    @classmethod
    def class_mapper(cls, d):
        for keys, cls in clsself.mappings.items():
            if keys.issuperset(d.keys()):   # are all required arguments present?
                return cls(**d)
        else:
            # Raise exception instead of silently returning None
            raise ValueError('Unable to find a matching class for object: {!s}'.format(d))

    @classmethod
    def complex_handler(cls, Obj):
        if hasattr(Obj, '__dict__'):
            return Obj.__dict__
        else:
            raise TypeError('Object of type %s with value of %s is not JSON serializable' % (type(Obj), repr(Obj)))

    @classmethod
    def register(cls, claz):
        clsself.mappings[frozenset(tuple([attr for attr,val in cls().__dict__.items()]))] = cls
        return cls

    @classmethod
    def to_json(cls, obj):
        return json.dumps(obj.__dict__, default=cls.complex_handler, indent=4)

    @classmethod
    def from_json(cls, json_str):
        return json.loads(json_str, object_hook=cls.class_mapper)

Usage:

@JsonConvert.register
class Employee(object):
    def __init__(self, Name:int=None, Age:int=None):
        self.Name = Name
        self.Age = Age
        return

@JsonConvert.register
class Company(object):
    def __init__(self, Name:str="", Employees:[Employee]=None):
        self.Name = Name
        self.Employees = [] if Employees is None else Employees
        return

company = Company("Contonso")
company.Employees.append(Employee("Werner", 38))
company.Employees.append(Employee("Mary"))

as_json = JsonConvert.to_json(company)
from_json = JsonConvert.from_json(as_json)
as_json_from_json = JsonConvert.to_json(from_json)

assert(as_json_from_json == as_json)

print(as_json_from_json)

Answer 14:

Expanding on DS’s answer a bit, if you need the object to be mutable (which namedtuple is not), you can use the recordclass library instead of namedtuple:

import json
from recordclass import recordclass

data = '{"name": "John Smith", "hometown": {"name": "New York", "id": 123}}'

# Parse into a mutable object
x = json.loads(data, object_hook=lambda d: recordclass('X', d.keys())(*d.values()))

The modified object can then be converted back to json very easily using simplejson:

x.name = "John Doe"
new_json = simplejson.dumps(x)

References