feature: `IntDecMode` `float64` and `json.Number` modes (`encoding/json` compatibility)

[extemporalgenome]

Is your feature request related to a problem? Please describe.

encoding/json's behavior when decoding arbitrary numbers to any is either float64, or json.Number, depending on decoder configuration.

fxamacker/cbor already has pretty good method-signature compatibility with encoding/json, but in the case of untyped decoding, requires additional complexity on the calling code in the form of additional types which must be handled: int64, uint64, and *big.Int.

Describe the solution you'd like

• Add an IntDecMode value to cause all CBOR integers (including *big.Int) to be decoded as float64 when the destination is *any. This would, of course, be lossy for values far from zero.
• Add an IntDecMode value to cause all CBOR integers (including *big.Int) to be decoded as json.Number. This would not be especially efficient, but would provide excellent compatibility with encoding/json for existing applications needing arbitrary-precision number support. Internally, this could just use behave like IntDecConvertNone with a post-process call to strconv.FormatInt, strconv.FormatUint, or (*big.Int).String, depending on the underlying type; or some specialized binary-to-decimal converter.
• Optional: JSONCompatDecOptions and JSONNumberDecOptions (or similar) functions, which set these options, as well as setting DefaultMapType to map[string]any.

[theory]

For those looking for solutions, this comment shows how to preserve json.Number values using the recently-assigned CBOR tag 284, the JSON Number String Tag.

[elv-gilles]

Even when activating support for JSON Number with tag 284, decoding into a struct that has a numeric field, e.g. int64, currently fails cannot unmarshal UTF-8 text string into Go struct field cbor_test.TimeSerial.update_time_epoch_sec of type int64

This would probably need to be handled in decoder.parseToValue.

Unit-test (click to expand)

package cbor_test

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io"
	"log"
	"reflect"
	"testing"
	"time"

	"github.com/fxamacker/cbor/v2"
)

type TimeSerial struct {
	//UpdateTime         time.Time `json:"update_time"`
	UpdateTimeEpochSec int64 `json:"update_time_epoch_sec"`
}

func makeCborCodec() (func(writer io.Writer) *cbor.Encoder, func(writer io.Reader) *cbor.Decoder) {
	var err error
	tagSet := cbor.NewTagSet()
	options := cbor.TagOptions{
		DecTag: cbor.DecTagOptional, 
		EncTag: cbor.EncTagRequired,
	}

	err = tagSet.Add(options, reflect.TypeOf(json.Number("")), 284)
	if err != nil {
		log.Fatal("invalid cbor tag", err, "tag", 284)
	}

	encOptions := cbor.CoreDetEncOptions()
	encOptions.Time = cbor.TimeUnixMicro
	encOptions.TimeTag = cbor.EncTagRequired
	enc, err := encOptions.EncModeWithTags(tagSet)
	if err != nil {
		log.Fatal("failed to create cbor encoder mode", err)
	}

	dec, err := cbor.DecOptions{
		DefaultMapType: reflect.TypeOf((map[string]interface{})(nil)),
	}.DecModeWithTags(tagSet)
	if err != nil {
		log.Fatal("failed to create cbor decoder mode", err)
	}

	return func(w io.Writer) *cbor.Encoder {
			return enc.NewEncoder(w)
		}, func(r io.Reader) *cbor.Decoder {
			return dec.NewDecoder(r)
		}
}

func TestSerialWithJsonNumber(t *testing.T) {
	now := time.Now()

	ts := &TimeSerial{
		//UpdateTime:         now,
		UpdateTimeEpochSec: now.Unix(),
	}
	var tsiFromJs interface{}
	{
		bb, err := json.Marshal(ts)
		if err != nil {
			t.Errorf("unexpected error %v", err)
		}
		dec := json.NewDecoder(bytes.NewReader(bb))
		dec.UseNumber() // preserve number formats
		err = dec.Decode(&tsiFromJs)
		if err != nil {
			t.Errorf("unexpected error %v", err)
		}
	}

	encFn, decFn := makeCborCodec()
	{
		buf := bytes.NewBuffer(make([]byte, 0))
		enc := encFn(buf)
		err := enc.Encode(&tsiFromJs)
		if err != nil {
			t.Errorf("unexpected error %v", err)
		}

		diag, err := cbor.Diagnose(buf.Bytes())
		if err != nil {
			t.Errorf("unexpected error %v", err)
		}
		fmt.Println(diag)

		dec := decFn(bytes.NewReader(buf.Bytes()))
		ts2 := &TimeSerial{}
		err = dec.Decode(ts2)
		if err != nil {
			t.Errorf("unexpected error %v", err)
		}
	}

}

[elv-gilles]

@fxamacker - a branch is ready here to further improves interop with json.

This covers cases where some json is received (e.g. on an API server), decoded via encoding/json before being cbor encoded. The json decoder can be configured with or without the UseNumber option of the json decoder.

The changes in the proposed branch:

• provide an 'opt-in' option to allow float to int conversion. This covers the case where json is decoded without the UseNumber option (and thus results in float values).
• adding support for JSON number (tag 284). This covers the case where the UseNumber option is used.

These changes do not modify the existing behavior since float-to-int is opt-in and support for JSON numbers has to be configured on the cbor encoder.

Please let me know if you are interested in a PR.

[benluddy]

I coincidentally implemented a FloatToInt option yesterday (without a lenient mode) in #777 and saw your backlink this morning (#439 (comment)). Is there any interest in spinning off the FloatToInt decode option from the proposed IntDecMode changes and json.Number support (@fxamacker @elv-gilles)?

It's not uncommon for programs in the Kubernetes ecosystem to using encoding/json to unmarshal a JSON representation of an API object from a file into interface{}, then serialize it into an HTTP request body. Since the stdlib unmarshals even integral-valued JSON numbers to float64, and CBOR preserves the distinction between integers and floating point numbers on the wire, the API server needs to be able to accept integral-valued CBOR floats when decoding them into int/uint fields.

[elv-gilles]

@benluddy - In my opinion, these changes go together but I feel a bit conflicted:

• on one hand, a FloatToInt option would already be a good first step (with lenient mode to help preventing random failures)
• on the other hand, configuring the API server with useNumber would look more like the real fix

[fxamacker]

Is there any interest in spinning off the FloatToInt decode option from the proposed IntDecMode changes and json.Number support (@fxamacker @elv-gilles)?

@benluddy Yes, I think FloatToInt would be useful to improve JSON and CBOR interoperability.

@elv-gilles Apologies for the late reply. I took a quick look at your branch and will need to take a closer look.