Serde

Bigflow 内置序列化/反序列化器

class bigflow.serde.BoolSerde

基类：bigflow.serde.CppSerde

bool类型序列化/反序列化器

• deserialize(buf)

• serialize(obj)

class bigflow.serde.CPickleSerde

基类：bigflow.serde.CppSerde

使用CPickle去做序列化/反序列化的Serde

class bigflow.serde.ChainSerde(*args)

基类：bigflow.serde.CppSerde

serde for chain transform. for example:

serdex should deserialize a string unless the last serde. ChainSerde(serdeA, serdB); serdeB->serialize(serdeA->serialize(object,obj)); serdeA->deserialize(serdeB->deserialize(buf));

• deserialize(buf)

• get_args()

• serialize(obj)

class bigflow.serde.CppSerde

基类：bigflow.serde.Serde

所有c++实现的serde类都要求继承于此类

• get_entity_config()

• get_entity_name()

  获取类名

class bigflow.serde.DefaultSerde

基类：bigflow.core.serde.omnitypes_objector.OmniObjector, bigflow.serde.CppSerde

默认的序列化/反序列化器

class bigflow.serde.DictSerde(key, value)

基类：bigflow.serde.CppSerde

Serde for dict

• deserialize(buf)

• serialize(obj)

class bigflow.serde.FastPbSerdeFactory(pipeline, proto_file)

基类：object

Create ProtobufSerde by fast-python-pb

>>> _serde_factory = serde.FastPbSerdeFactory(_pipeline, './sample.proto')
>>> _serde = _serde_factory.get_serde('bigflow.MESSAGE')
>>> data = _pipeline.read(input.SequenceFile(input_path, serde=_serde))

• get_serde(class_name)

  get protobuf serde

class bigflow.serde.FloatSerde

基类：bigflow.serde.CppSerde

float类型序列化/反序列化器

• deserialize(buf)

• serialize(obj)

class bigflow.serde.IdlPacketSerde(log_type)

基类：bigflow.serde.CppSerde

Serde for idl_packet

• deserialize(buf)

• get_log_type()

  get log_type

• serialize(obj)

  serialzie

class bigflow.serde.IntSerde

基类：bigflow.serde.CppSerde

int类型序列化/反序列化器

• deserialize(buf)

• serialize(obj)

  serialize int obj

class bigflow.serde.ListSerde(*value)

基类：bigflow.serde.CppSerde

list序列化/反序列化器

• get_fields()

class bigflow.serde.ObjectorSerde(name, config=’’)

基类：bigflow.serde.Serde

Serde for objector serde

• get_entity_config()

• get_entity_name()

  获取类名

class bigflow.serde.Optional(serde)

基类：bigflow.serde.CppSerde

用于支持None的序列化/反序列化

• deserialize(buf)

• origin_serde()

• serialize(obj)

class bigflow.serde.ProtobufSerde(msg_type, raise_exception_when_failed=True)

基类：bigflow.serde.Serde

Serde for protobuf

• deserialize(buf)

• get_msg_type()

• serialize(msg)

class bigflow.serde.SameTypeListSerde(value)

基类：bigflow.serde.CppSerde

list序列化/反序列化器，要求list中元素均为同一类型

• deserialize(buf)

• serialize(obj)

class bigflow.serde.Serde

基类：bigflow.core.entity.EntitiedBySelf

所有python实现的serde类要求继承于此类，用户可以通过继承Serde实现自定义的序列化/反序列化器

>>> from bigflow import serde
>>>
>>> class CustomSerde(serde.Serde):
...     def serialize(self, obj):
...         assert(isinstance(obj, MyCostomObject))
...         return str(obj)
...     def deserialize(self, buf):
...         return MyCostomObject(buf)

在大部分的变换中都可以传递一个特殊的叫serde的参数， 设置该参数即可指定产出的PCollection的serde。

p.map(lambda x: MyCostomObject(x), serde=CustomSerde())

• get_entity_config()

• get_entity_name()

  获取类名

class bigflow.serde.SetSerde(value)

基类：bigflow.serde.CppSerde

Serde for set

• deserialize(buf)

• get_value_serde()

• serialize(obj)

class bigflow.serde.StrSerde

基类：bigflow.serde.CppSerde

str类型序列化/反序列化器

• deserialize(buf)

• serialize(obj)

class bigflow.serde.TupleLikeListSerde(*value)

基类：bigflow.serde.CppSerde

list序列化/反序列化器，list中每个元素可以为任意类型

• deserialize(buf)

• serialize(obj)

class bigflow.serde.TupleSerde(*args)

基类：bigflow.serde.CppSerde

tuple序列化/反序列化器

• deserialize(buf)

• get_args()

• serialize(obj)

bigflow.serde.any()

Return a serde use the default serde.

bigflow.serde.bool_()

Return an optional bool serde.

bigflow.serde.common_serde(*serdes)

输出输入serde集合中的公共serde 目前只维护系统定义的serde, 对于用户自定义的serde, 返回None :param serdes: serde instances or classes

bigflow.serde.dict_of(key, value)

Return an optional dict serde

bigflow.serde.extract_elem(serde, n, if_not_found=None)

inner function

bigflow.serde.float_()

Return an optional float serde.

bigflow.serde.int_()

Return an optional int serde.

bigflow.serde.list_of(*value)

Return an optional list serde.

bigflow.serde.of(rtype)

Return the serde you want.

If the input arg is a type instance, such as int, str, float or a tuple of type instances, such as (int, str) or a list of type instances, such as [int, str] and [int] the function will return the corresponding serde to serialize and deserialize the data.

The returned serde can process the Nonable data.

• Note: [int] means a list of ints, it can accept any number of ints you want,

  but [int, str] can only accecpt a list who has exactly 2 elements. (if your data to serialize has more than 2 elements, the remaining elements may be lost)

bigflow.serde.origin(serde)

inner function

bigflow.serde.proto_of(msg_type)

Return an protobuf serde Warning: this is not optional

bigflow.serde.sample(value)

Return a serde can process your input data.

Note: if the sample is a list, when all the data is the same type, we assume

bigflow.serde.set_of(value)

Return an optional set serde

bigflow.serde.str_()

Return an optional str serde.

bigflow.serde.tuple_of(*args)

Return an optional tuple serde.