python tempfile + gzip + json dump

Amir :

I want to dump very large dictionary in to a compressed json file using python3 (3.5).

import gzip
import json
import tempfile

data = {"verylargedict": True}

with tempfile.NamedTemporaryFile("w+b", dir="/tmp/", prefix=".json.gz") as fout:
    with gzip.GzipFile(mode="wb", fileobj=fout) as gzout:
        json.dump(data, gzout)

I got this error though.

Traceback (most recent call last):
  File "test.py", line 13, in <module>
    json.dump(data, gzout)
  File "/usr/lib/python3.5/json/__init__.py", line 179, in dump
    fp.write(chunk)
  File "/usr/lib/python3.5/gzip.py", line 258, in write
    data = memoryview(data)
TypeError: memoryview: a bytes-like object is required, not 'str'

Any thoughts?

Jean-François Fabre :

Gzip object has no text mode. So I would create a wrapper to pass as the filehandle object. This wrapper takes data from json and encodes it as binary to write in the gzip file:

class wrapper:
    def __init__(self,gzout):
        self.__handle = gzout
    def write(self,data):
        self.__handle.write(data.encode())

use like this:

json.dump(data, wrapper(gzout))

each time json.dump wants to write to the object, the wrapper.write method is called, which converts text to binary and writes to the binary stream

(some built-in wrappers from io module may fit too, but this implementation is simple and work)

Guess you like

Origin http://43.154.161.224:23101/article/api/json?id=279381&siteId=1

python tempfile + gzip + json dump

Python tempfile + gzip + json Dump

python tempfile + gzip + despejo json

python tempfile + gzip + vidage JSON

python tempfile creates a temporary directory

Difference between json.dump () and json.dumps () in python

Python serialization (json.dump()) and deserialization (json.load()) of files

pitón tempfile + gzip + JSON vertedero

What is serialized, Python in the json load, loads, dump, dumps and pickle the load, loads, dump, dumps the difference

Python Json the load, loads, dump, dumps and operations analysis

python json module dumps, dump, loads, load function describes

Python json library load, loads, dump, dumps the difference and usage

python tempfile module --- generate temporary files and directories

About json dump and dumps

The difference .json and json.dump

Python reads and writes json files (dump, load), and processes data in json format (dumps, loads)

Python - DIY - Use dump to take some key-value pairs from json to synthesize a new json file

Understand Python's json_dump(s) and json_load(s) in one article

[Python self-study notes] The difference between json.loads and json.load (json.dumps and json.dump)

python tempfile module: generate temporary files and temporary directories

Python数据分析【第5天】| 读、写json文件（load()、loads()、dump()、dumps()）

Python modules using the gzip

json -- dump load dumps loads simple comparison

python import gzip use analytic

[Algorithm] Use GZIP compression algorithm to compress the transmitted JSON string

python post submit gzip form data

Difference in python dump, load and dumps, loads of

python to fetch data dump oracle mysql

python excel table and reads the data dictionary as a dump

Python-like dumps / loads / dump / load