Server limited to 25 characteristics

[MrCavemen]

I'm trying to run a bridge with 14 blinds (as different accessories as I want to put them in different rooms).
These each have 3 services.

My setup can handle 9 blinds, from 10 blinds and above, the accessories have the red "not responsive" text.
Output from the terminal tells me that when all target and position characteristics are being requested not all of them "make it".
For only 12/14 blinds, both target and current position is being asked. For the unlucky 13th, only one is asked, and the 14th blind is ignored.

Fun shenanigans:

• (They are in a bridge, as it seems that when you have more than 8 accessories, homekit automatically assumes that it is a bridge. When you don't add the bridge, the first accessory will be seen as the bridge, and you will not be able to interact with it.)
• When you add the bridge to the home hub, each one of them does update to the closed position, but after the adding steps, they are all listed as unresponsive.
• Long pressing an accessory allows one to change the blind position, as it temporarily becomes responsive (until the home_hub requests all characteristics).
• Siri can also change the position of these unresponsive blinds (as they temporarily become responsive)
• Official homekit accessories (two smoke detectors) can also not update (become briefly unresponsive) when requesting all characteristics

My Setup:

• Hub: Original iPad Air (iOS 12.5.5, <- latest supported software version)
• ESP32 (4MB)
• esp-idf v4.3.2
• maximkulkin/esp-homekit and demo: latest commits

Part of the main.c code:

// ext := extension to separate pcb boards
// n := blind number on pcb
#define gen_blinds_accessory(ext, n, idd) \
HOMEKIT_ACCESSORY(.id= idd ,\
    .category=homekit_accessory_category_window_covering,\
    .services=(homekit_service_t*[]){\
        HOMEKIT_SERVICE(ACCESSORY_INFORMATION,\
            .characteristics=(homekit_characteristic_t*[]){\
                HOMEKIT_CHARACTERISTIC(NAME, "Blind"#ext"_"#n),\
                HOMEKIT_CHARACTERISTIC(MANUFACTURER, "Me"),\
                HOMEKIT_CHARACTERISTIC(SERIAL_NUMBER, "B"#ext"_"#n),\
                HOMEKIT_CHARACTERISTIC(MODEL, "BLIND"),\
                HOMEKIT_CHARACTERISTIC(FIRMWARE_REVISION, "0.1"),\
                HOMEKIT_CHARACTERISTIC(IDENTIFY, NULL),\
                NULL\
        }),\
    HOMEKIT_SERVICE(WINDOW_COVERING, .primary=true,\
        .characteristics=(homekit_characteristic_t*[]){\
            &blind_current_pos##ext##_##n,\
            &blind_target_pos##ext##_##n,\
            &blind_pos_state##ext##_##n,\
            NULL\
        }),\
    NULL}),



homekit_accessory_t *accessories[] = {
    HOMEKIT_ACCESSORY(.id=1 ,
        .category=homekit_accessory_category_bridge,
        .services=(homekit_service_t*[]){
            HOMEKIT_SERVICE(ACCESSORY_INFORMATION,
                .characteristics=(homekit_characteristic_t*[]){
                    HOMEKIT_CHARACTERISTIC(NAME, "CU Bridge"),
                    HOMEKIT_CHARACTERISTIC(MANUFACTURER, "Me"),
                    HOMEKIT_CHARACTERISTIC(SERIAL_NUMBER, "B"),
                    HOMEKIT_CHARACTERISTIC(MODEL, "BRIDGE"),
                    HOMEKIT_CHARACTERISTIC(FIRMWARE_REVISION, "0.1"),
                    HOMEKIT_CHARACTERISTIC(IDENTIFY, NULL),
                    NULL
                }),
        NULL}),
    gen_blinds_accessory(_1,2,2)
    gen_blinds_accessory(_1,3,3)
    gen_blinds_accessory(_1,4,4)
    gen_blinds_accessory(_1,6,5)
    gen_blinds_accessory(_1,7,6)
    gen_blinds_accessory(_1,8,7)

    gen_blinds_accessory(_2,1,8)
    gen_blinds_accessory(_2,2,9)
    gen_blinds_accessory(_2,3,10)
    gen_blinds_accessory(_2,4,11)
    gen_blinds_accessory(_2,5,12)
    gen_blinds_accessory(_2,6,13)
    gen_blinds_accessory(_2,7,14)
    gen_blinds_accessory(_2,8,15)
    NULL
};

Part of the blinds.c code:

//TODO store on FLASH memory to survive reboots & power outages
struct BLIND_DATA {
    uint8_t current;
    uint8_t target;
    uint8_t pos_state;
    bool hold;
};
static struct BLIND_DATA data_1[8];
static struct BLIND_DATA data_2[8];

#define gen_c_CURRENT_POS(ext, n) homekit_characteristic_t blind_current_pos##ext##_##n = \
    HOMEKIT_CHARACTERISTIC_(CURRENT_POSITION, \
                            100, \
                            .getter_ex=get_HK_UINT8_value, \
                            .setter_ex=set_HK_UINT8_value, \
                            .context=(void*)&(data##ext[n-1].current));

homekit_value_t get_HK_UINT8_value(const homekit_characteristic_t *ch) {
    uint8_t *data = (uint8_t*) ch->context;
    printf("printing get HK value for blind ~~~~~~~~~~~~~~\n");
    printf("uint8 data: %i\n", *data);
    return HOMEKIT_UINT8(*data);
}
void set_HK_UINT8_value(homekit_characteristic_t *ch, const homekit_value_t value) {
    printf("printing HK value for blind ~~~~~~~~~~~~~~\n");
    printf("uint8 value: %i\n", value.uint8_value);
    printf("uint  value: %i\n", value.int_value);
    
    uint8_t *data = ch->context;
    *data = value.uint8_value;
}

Output log:
(Note that for only 12/14 blinds, both target and current position is being asked. For the unlucky 13th, only one is asked, and the 14th blind is ignored.)

>>> HomeKit: [Client 2] Get Characteristics
>>> homekit_server_on_get_characteristics: Free heap: 185608
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 10.9
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 10.10
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 13.9
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 13.10
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 8.10
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 8.9
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 6.9
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 6.10
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 12.10
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 12.9
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 5.9
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 5.10
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 2.9
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 2.10
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 7.10
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 7.9
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 14.10
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 14.9
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 3.9
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 3.10
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 11.10
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 11.9
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 9.10
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 9.9
>>> homekit_server_on_get_characteristics: [Client 2] Requested characteristic info for 4.10
>>> client_sendv: [Client 2] Sending payload: HTTP/1.1 200 OK\x0D\x0AContent-Type: application/hap+json\x0D\x0ATransfer-Encoding: chunked\x0D\x0AConnection: keep-alive\x0D\x0A\x0D\x0A
>>> HomeKit: [Client 2] Requested characteristic info for 10.9 ("Current Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> HomeKit: [Client 2] Requested characteristic info for 10.10 ("Target Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> HomeKit: [Client 2] Requested characteristic info for 13.9 ("Current Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> HomeKit: [Client 2] Requested characteristic info for 13.10 ("Target Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> HomeKit: [Client 2] Requested characteristic info for 8.10 ("Target Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> HomeKit: [Client 2] Requested characteristic info for 8.9 ("Current Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> HomeKit: [Client 2] Requested characteristic info for 6.9 ("Current Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> HomeKit: [Client 2] Requested characteristic info for 6.10 ("Target Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> client_sendv: [Client 2] Sending payload: 3ff\x0D\x0A{"characteristics":[{"aid":10,"iid":9,"description":"Current Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":10,"iid":10,"description":"Target Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":13,"iid":9,"description":"Current Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":13,"iid":10,"description":"Target Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":8,"iid":10,"description":"Target Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":8,"iid":9,"description":"Current Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":6,"iid":9,"description":"Current Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":6,"iid":10,"description":"\x0D\x0A
>>> HomeKit: [Client 2] Requested characteristic info for 12.10 ("Target Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> HomeKit: [Client 2] Requested characteristic info for 12.9 ("Current Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> HomeKit: [Client 2] Requested characteristic info for 5.9 ("Current Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> HomeKit: [Client 2] Requested characteristic info for 5.10 ("Target Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> HomeKit: [Client 2] Requested characteristic info for 2.9 ("Current Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> HomeKit: [Client 2] Requested characteristic info for 2.10 ("Target Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> HomeKit: [Client 2] Requested characteristic info for 7.10 ("Target Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> client_sendv: [Client 2] Sending payload: 3ff\x0D\x0ATarget Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":12,"iid":10,"description":"Target Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":12,"iid":9,"description":"Current Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":5,"iid":9,"description":"Current Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":5,"iid":10,"description":"Target Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":2,"iid":9,"description":"Current Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":2,"iid":10,"description":"Target Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":7,"iid":10,"description":"Target Position","format":"uint8","unit":"percentage","\x0D\x0A
>>> HomeKit: [Client 2] Requested characteristic info for 7.9 ("Current Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> HomeKit: [Client 2] Requested characteristic info for 14.10 ("Target Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> HomeKit: [Client 2] Requested characteristic info for 14.9 ("Current Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> HomeKit: [Client 2] Requested characteristic info for 3.9 ("Current Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> HomeKit: [Client 2] Requested characteristic info for 3.10 ("Target Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> HomeKit: [Client 2] Requested characteristic info for 11.10 ("Target Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> HomeKit: [Client 2] Requested characteristic info for 11.9 ("Current Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> HomeKit: [Client 2] Requested characteristic info for 9.10 ("Target Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> client_sendv: [Client 2] Sending payload: 400\x0D\x0AminValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":7,"iid":9,"description":"Current Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":14,"iid":10,"description":"Target Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":14,"iid":9,"description":"Current Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":3,"iid":9,"description":"Current Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":3,"iid":10,"description":"Target Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":11,"iid":10,"description":"Target Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":11,"iid":9,"description":"Current Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"ai\x0D\x0A
>>> HomeKit: [Client 2] Requested characteristic info for 9.9 ("Current Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> HomeKit: [Client 2] Requested characteristic info for 4.10 ("Target Position")
printing get HK value for blind ~~~~~~~~~~~~~~
uint8 data: 0
>>> client_sendv: [Client 2] Sending payload: 19b\x0D\x0Ad":9,"iid":10,"description":"Target Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":9,"iid":9,"description":"Current Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0},{"aid":4,"iid":10,"description":"Target Position","format":"uint8","unit":"percentage","minValue":0,"maxValue":100,"minStep":1,"value":0}]}\x0D\x0A
>>> client_sendv: [Client 2] Sending payload: \x30\x0D\x0A\x0D\x0A
>>> homekit_client_process: [Client 2] Finished processing

[maccoylton]

What chip are you using? Devices like the esp8266 have limited memory, each service & characteristic consumes some memory, you may be hitting the limit of what the chip can handle.

[MrCavemen]

I'm running the ESP-wroom-32. I've attached the details from the chip below
Before the requests, the Free heap was 185608. So it doesn't seems to be a memory issue.

To be absolutely sure, I now also print the free heap when the characteristics are requested.
The lowest I see is 189804. (I've reduced my memory footprint a bit since last time.)

I (27) boot: ESP-IDF v4.3.2 2nd stage bootloader
I (27) boot: compile time 16:29:17
I (27) boot: chip revision: 1
I (30) boot_comm: chip revision: 1, min. bootloader chip revision: 0
I (37) boot.esp32: SPI Speed      : 40MHz
I (41) boot.esp32: SPI Mode       : DIO
I (46) boot.esp32: SPI Flash Size : 4MB
I (50) boot: Enabling RNG early entropy source...
I (56) boot: Partition Table:
I (59) boot: ## Label            Usage          Type ST Offset   Length
I (67) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (74) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (82) boot:  2 factory          factory app      00 00 00010000 00100000
I (89) boot: End of partition table
I (93) boot_comm: chip revision: 1, min. application chip revision: 0
I (100) esp_image: segment 0: paddr=00010020 vaddr=3f400020 size=1dcach (122028) map
I (153) esp_image: segment 1: paddr=0002dcd4 vaddr=3ffb0000 size=02344h (  9028) load
I (157) esp_image: segment 2: paddr=00030020 vaddr=400d0020 size=95150h (610640) map
I (380) esp_image: segment 3: paddr=000c5178 vaddr=3ffb2344 size=042e4h ( 17124) load
I (387) esp_image: segment 4: paddr=000c9464 vaddr=40080000 size=1671ch ( 91932) load
I (425) esp_image: segment 5: paddr=000dfb88 vaddr=50000000 size=00010h (    16) load
I (437) boot: Loaded app from partition at offset 0x10000
I (437) boot: Disabling RNG early entropy source...
I (448) cpu_start: Pro cpu up.
I (448) cpu_start: Starting app cpu, entry point is 0x400813b8
0x400813b8: call_start_cpu1 at /HomeKit/esp-idf/components/esp_system/port/cpu_start.c:150

I (0) cpu_start: App cpu up.
I (463) cpu_start: Pro cpu start user code
I (463) cpu_start: cpu freq: 160000000
I (463) cpu_start: Application information:
I (467) cpu_start: Project name:     led
I (472) cpu_start: App version:      f8dcf29-dirty
I (477) cpu_start: Compile time:     Jan  5 2022 12:37:16
I (483) cpu_start: ELF file SHA256:  f84209b702d7d407...
I (489) cpu_start: ESP-IDF:          v4.3.2
I (494) heap_init: Initializing. RAM available for dynamic allocation:
I (501) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (507) heap_init: At 3FFBAC38 len 000253C8 (148 KiB): DRAM
I (514) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (520) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (526) heap_init: At 4009671C len 000098E4 (38 KiB): IRAM
I (534) spi_flash: detected chip: generic
I (537) spi_flash: flash io: dio
I (542) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
I (606) wifi:wifi driver task: 3ffbc66c, prio:23, stack:6656, core=0
I (606) system_api: Base MAC address is not set
I (606) system_api: read default base MAC address from EFUSE
I (616) wifi:wifi firmware version: eb52264
I (616) wifi:wifi certification version: v7.0
I (616) wifi:config NVS flash: enabled
I (616) wifi:config nano formating: disabled
I (626) wifi:Init data frame dynamic rx buffer num: 32
I (626) wifi:Init management frame dynamic rx buffer num: 32
I (636) wifi:Init management short buffer num: 32
I (636) wifi:Init dynamic tx buffer num: 32
I (646) wifi:Init static rx buffer size: 1600
I (646) wifi:Init static rx buffer num: 10
I (646) wifi:Init dynamic rx buffer num: 32

[MrCavemen]

I found the "issue".
In the server, the endpoint_params has a hard coded ids array of 25.
When having more than 25 characteristics, the characteristics beyond 25 are ignored.
This has a side effect causing none of the characteristics to properly update.

For my use case, I simply increased this to 100. (Causing 300 bytes of extra memory consumption)

To ensure a low memory profile for the esp8266, maybe this should be made dynamic?
We can store the id_count from homekit_server_on_url, store a pointer in the endpoint_params an realloc it's memory when needed.
Alternatively, it could be made user configurable from the menuconfig, but this seems to be less user-friendly.

[maximkulkin]

So, yes, I agree, that it can be made configurable with ESP32 devices getting larger values by default plus allowing overriding defaults if really needed. Will think about automating it (e.g. automatically configuring it to the number of characteristics present).