Ticket #90 (new enhancement)

Opened 8 years ago

Last modified 5 years ago

Allow TellStick Duo to receive new protocols

Reported by: http://login.telldus.com/id?i=10&h=174530911a9afc527ca9880c9e6c75cca0d32fe2 Owned by: micke prag <micke.prag@…>
Priority: minor Milestone: Some future release / Patches welcome
Component: other Version:
Keywords: Cc:
Blocked By: Blocking:
Platform: Other Sensitive: no
Verified by Telldus: no

Description

Possibility to, via software, set TellStick Duo in "promiscuous" mode, where it just filters out garbage, but all that could be valid signals are sent on to the computer. This would allow TellStick Duo owners to participate in adding support for more protocols.

Change History

comment:1 Changed 8 years ago by micke prag <micke.prag@…>

  • Owner changed from mprag to micke prag <micke.prag@…>
  • Status changed from new to assigned

comment:2 Changed 7 years ago by micke prag <micke.prag@…>

  • Status changed from assigned to new
  • Verified by Telldus unset

comment:3 Changed 6 years ago by jyrki kuoppala <jkp@…>

I tested something like this to add support for a new protocol (with a modified fineoffset file). I added some extra data outpout for similar bit counts, which helped me to spot patterns of the lengths of the encoding. The code is currently very experimental, so I'm not sure if it tells what's happening that well, but the basic approach was to have some durations of 0-strings and 1-strings (bits of contiguous ones and zeros) as extra debug data in the "data" hex string. (the commented-out DATA_LENGTH*3 at the end), which are then passed to the host. The way this is done with the experimental code requires also changing telldus-core, instead could perhaps have a separate mode where user could subscribe to protocol-specific or generic extra data arrays.

* ../../tellstick-duo/rf/fineoffset.c 2013-01-03 16:53:47.000000000 +0200
--- fineoffset.c 2013-01-05 00:06:52.000000000 +0200

#define SHORT(x) (x<11)
#define LONG(x) (x>13)


#define INVALID_DATA 2

! #define DATA_LENGTH 5

! unsigned char fineOffsetBit(unsigned short *scanP, unsigned char *scanBit) {

UCHAR8 b1 = rfCountSimilar(scanP, scanBit);
UCHAR8 b2 = rfCountSimilar(scanP, scanBit);


if (LONG(b1) && LONG(b2)) {

! return 0;

}
if (SHORT(b1) && LONG(b2)) {

! return 1;

}


return INVALID_DATA;

}


void parseFineOffset(unsigned short scanP, unsigned char scanBit) {

! unsigned char buffer[DATA_LENGTH];

rfRetreatBit(&scanP, &scanBit); skip last bit


for(int i=DATA_LENGTH-1; i>=0; --i) {

UCHAR8 byte = 0;
for(int j=0; j<8; ++j){

! UCHAR8 b = fineOffsetBit(&scanP, &scanBit);

if (b == INVALID_DATA) {

return;

}

! if (b) {

byte |= (1<<j);

}

}
buffer[i] = byte;

}
rfMessageBeginRaw();

rfMessageAddString("class", "sensor");
rfMessageAddString("protocol", "fineoffset");
rfMessageAddHexString("data", buffer, DATA_LENGTH);

rfMessageEnd(1);

}

--- 7,69 ----

#define SHORT(x) (x<11)
#define LONG(x) (x>13)

+ #define LONGER(x) (x>22)
+ #define SHORT(x) (x<14)
+
#define LONG(x) (x>16)

#define INVALID_DATA 2

! #define DATA_LENGTH 10

! unsigned int fineOffsetBit(unsigned short *scanP, unsigned char *scanBit) {

UCHAR8 b1 = rfCountSimilar(scanP, scanBit);
UCHAR8 b2 = rfCountSimilar(scanP, scanBit);


+ if (LONG(b1) && LONGER(b2)) {
+ if (LONG(b1) && LONGER(b2)) {
+ return 1 + b1 * 256 * 16 + b2*2;
+ }

if (LONG(b1) && LONG(b2)) {

! return 0 + b1 * 256 * 16 + b2*2;

}
if (SHORT(b1) && LONG(b2)) {

! return 0 + b1 * 256 * 16 + b2*2;
! }
! if (SHORT(b1) && SHORT(b2)) {
! return INVALID_DATA;

}


return INVALID_DATA;

}


void parseFineOffset(unsigned short scanP, unsigned char scanBit) {

! unsigned char buffer[DATA_LENGTH*3];

rfRetreatBit(&scanP, &scanBit); skip last bit


for(int i=DATA_LENGTH-1; i>=0; --i) {

UCHAR8 byte = 0;

+ int b = 0;

for(int j=0; j<8; ++j){

! b = fineOffsetBit(&scanP, &scanBit);

if (b == INVALID_DATA) {

return;

}

! if ((b & 1) != 0) {

byte |= (1<<j);

}

+ if ((b & 2047) / 2 > 20) {
+ byte |= (1<<j);
+ }
+ if (j == 0) {
+ buffer[DATA_LENGTH+i] = (b / 256 / 16);
+ buffer[DATA_LENGTH*2+i] = (b & 2047) / 2;
+ }

}
buffer[i] = byte;

}
rfMessageBeginRaw();

rfMessageAddString("class", "sensor");
rfMessageAddString("protocol", "fineoffset");

+ rfMessageAddHexString("data", buffer, DATA_LENGTH*3);

rfMessageAddHexString("data", buffer, DATA_LENGTH);

rfMessageEnd(1);

}

Last edited 6 years ago by jyrki kuoppala <jkp@…> (previous) (diff)

comment:4 Changed 6 years ago by jyrki kuoppala <jkp@…>

The problem is recognizing what's garbage and what's interesting. Kind of a chicken-and-egg problem - if we do have no idea of the protocol, we can't easily know where the noise ends and signal starts.

What helps is that the radio traffic sets some boundaries - sending zeros and ones directly won't produce good results on the radio channel, so there's some kind of coding. Looks like using pulses with width coding is one common approach, used e.g. in the fineoffset protocol one pulse length signifies a zero, another signifies a one. Manchester encoding is another.

One common approach to grokking 433Mhz signals seems to be visualizing the signal, by connecting the amplitude output of a radio receiver to an oscilloscope, or, for people without a scope, to a computer sound card with suitable software (like Audacity) working as an oscilloscope. This may allow recognizing different radio sources by the amplitude. But this probably can't be done with the Tellstick (nicely anyway, without breaking the case anyway - probably can't be done without tapping to the hardware if even then, and Micke.Prag tells us that if the Duo is opened it can't be put back together, no screws apparently)

Thinking of how to try to grok protocols with the Tellstick, outputting the reception transitions from one to zero and zero to one by lengths helped to recognize the protocol which the code above reads (see  http://www.telldus.se/forum/viewtopic.php?f=12&t=3086&sid=2ab0b664faceaf92a4ef8a951c6c8229 ). With some suitable computer-side software, this too could be visualized, helping to recognize the start of a packet and end of a packet, and get some idea if the signal is pulse width modulation, Manchester coding or something else.

If the packets have enough repetition (like probably most devices which send data do), it would allow for start and stop patterns to be set, or alternatively just the start pattern, and then a bit count.

So, for firmware to help debug unknown protocols, one approach would be two phases

1) add promiscuous mode to firmware: transmit everything to the computer, e.g. as zero string / one string lengths. Allow some way of the data to be represented textually and visually to help try to find how a packet start can be recognized

2) add triggered recording of a packet to firmware: a command specifies the start packet trigger bit pattern, and then alternatively either a packet size (in bits or bit transitions), or an end pattern. Then only packets matching the trigger (filter) are sent to the computer

Seems that in practice, at least with pulse width encoding, start packet would need to be specified with a pulse pattern with upper & lower bounds for each pulse length.

Last edited 6 years ago by jyrki kuoppala <jkp@…> (previous) (diff)

comment:5 Changed 6 years ago by jyrki kuoppala <jkp@…>

Might be easier to recognize a packet ending, so a trigger / filter with a packet ending described by bit transition lengths and a packet size would also be useful.

For the generic goal of allowing the Duo to receive new protocols, one approach would be to change the firmware radio reception architecture a bit, so the firmware wouldn't need to know the higher-level protocol details (e.g. packet lengths) of each device like it does now, but the higher level would be handled at telldus-core.

What I'm thinking of is that the computer would tell the firmware something like "I want to listen to PWM-encoded signals with pulse lengths between 28 and 34 for ones and 14 and 18 for zeros, with packet size between 10 and 20 and start & end described here." The firmware would then blink the leds & send the computer/router/whatever only the data which the user wants to subscribe to. This would cut down on spurious blinkenlichts and unnecessary traffic between Tellstick & host (like neigbour's sensors), and make things easier to grok for the end user. Telldus-core could of course also have a "full listen" mode so the user could look at all the data available to find out what are the wanted sensors, but the normal mode of operation would not list everything. With things set up like this, adding support e.g. for a rain or wind sensor might not require a new firmware, just an upgrade to telldus-core, if the low-level encoding is one already supported by the firmware.

Last edited 6 years ago by jyrki kuoppala <jkp@…> (previous) (diff)

comment:6 Changed 6 years ago by micke prag <micke.prag@…>

  • Sensitive unset

This is very interesting thoughts. Do you have a proof of concept implemetation of such firmware for TellStick Duo?
It would be interested how this will perform compared to the old code.

comment:7 Changed 6 years ago by jyrki kuoppala <jkp@…>

No, no proof of concept yet at least. The current firmware code I have is below as a diff, just addition of optional debug data (TIMINGS) and listening of all packets (PROMISCOUS), not even sure if that currently works for PROMISCUOUS. It does work with TIMINGS for the "Nexus" sensor (though currently requires a modified fineoffset file in telldus-core).

To move forward, could be a good idea to define the commands to modify the settings of the firmware. Protocol seems to be commands are single chars, so how about O command for Options, and parameter R for reception? Then perhaps something like:

OR- - turn off the current "receive all known protocols" behaviour
OR* - turn on promiscuous mode
OR+P1 - turn on reception of pulse-width encoding format #1
OR+M1 - turn on reception of Manchester encoding format #1

Most commonly used PWM / Manchester / whatever formats could possibly be at first hard-coded in the firmware (for best performance), maybe later add support for custom encodings something like:

ORFP50=20-30/10-15,10-30 - set custom pulse width encoding #50, where length of ones is between 20-30 time units, length of zeros is 10-15 time units, and packet length is between 10 and 30 bytes.
ORFP51=20-30/10-150000,FFFF - set custom pulse width encoding #50, where length of ones is between 20-30 time units, length of zeros is 10-15 time units, any packet length,packet starting pattern 0000, packet ending pattern FFFF
ORFP51=20-30/10-15
PT1,PT2 - set custom pulse width encoding #50, where length of ones is between 20-30 time units, length of zeros is 10-15 time units, any packet length,packet starting pattern #1, packet ending pattern #2 (allowing for starting patterns not part of data but encoded as out-of band bit patterns)

For the timings / raw pulse length debug output (TIMINGS code), instead of piggybacking the debug data after the "data" field, would make more sense to add a new field, "lengths", "raw", "rawpulses", or whatever would be understandable.

diff -cr tellstick-duo/rf/fineoffset.c tellstick-duo.jkp/rf/fineoffset.c
* tellstick-duo/rf/fineoffset.c 2013-01-03 16:53:47.000000000 +0200
--- tellstick-duo.jkp/rf/fineoffset.c 2013-01-08 01:15:12.000000000 +0200
*
* 5,51

#include <stdio.h>
#include <stdlib.h>


#define SHORT(x) (x<11)
#define LONG(x) (x>13)


#define INVALID_DATA 2

! #define DATA_LENGTH 5

! unsigned char fineOffsetBit(unsigned short *scanP, unsigned char *scanBit) {

UCHAR8 b1 = rfCountSimilar(scanP, scanBit);
UCHAR8 b2 = rfCountSimilar(scanP, scanBit);


! if (LONG(b1) && LONG(b2)) {
! return 0;

}
if (SHORT(b1) && LONG(b2)) {

! return 1;

}

!
! return INVALID_DATA;

}


void parseFineOffset(unsigned short scanP, unsigned char scanBit) {

unsigned char buffer[DATA_LENGTH];


rfRetreatBit(&scanP, &scanBit); skip last bit


for(int i=DATA_LENGTH-1; i>=0; --i) {

UCHAR8 byte = 0;
for(int j=0; j<8; ++j){

! UCHAR8 b = fineOffsetBit(&scanP, &scanBit);

if (b == INVALID_DATA) {

return;

}

! if (b) {

byte |= (1<<j);

}

}
buffer[i] = byte;

}
rfMessageBeginRaw();

rfMessageAddString("class", "sensor");
rfMessageAddString("protocol", "fineoffset");
rfMessageAddHexString("data", buffer, DATA_LENGTH);

rfMessageEnd(1);

}

--- 5,112 ----

#include <stdio.h>
#include <stdlib.h>


+ #define NEXUS
+ #define SHORT(x) (x<6)
+
#define LONG(x) (x>6)
+ #define SHORT(x) (x<14)
+
#define LONG(x) (x>16)
+ #else

#define SHORT(x) (x<11)
#define LONG(x) (x>13)

+ #endif
+ #define LONGER(x) (x>22)
+ #define VERYLONG(x) (x>128)

#define INVALID_DATA 2

! #ifdef NEXUS
! #define DATA_LENGTH 10
! #else
! #define DATA_LENGTH 19
! #endif
! #define TIMINGS
! #define PROMISCOUS
!
#define DATA_LENGTH 60

! unsigned int fineOffsetBit(unsigned short *scanP, unsigned char *scanBit) {

UCHAR8 b1 = rfCountSimilar(scanP, scanBit);
UCHAR8 b2 = rfCountSimilar(scanP, scanBit);


! #ifdef PROMISCOUS
! if (VERYLONG(b1)) {
! return INVALID_DATA;
! }
! if (LONG(b2)) {
! return 1 + b1 * 256 * 16 + b2*2;
! }
! if (SHORT(b2)) {
! return 0 + b1 * 256 * 16 + b2*2;

}
if (SHORT(b1) && LONG(b2)) {

! return 0 + b1 * 256 * 16 + b2*2;

}

! return 0 + b1 * 256 * 16 + b2*2;
! #else
! if (LONG(b1) && LONGER(b2)) {
! return 1 + b1 * 256 * 16 + b2*2;
! }
! if (LONG(b1) && LONG(b2)) {
! return 0 + b1 * 256 * 16 + b2*2;
! }
! if (SHORT(b1) && LONG(b2)) {
! return 0 + b1 * 256 * 16 + b2*2;
! }
! if (SHORT(b1) && SHORT(b2)) {
! return INVALID_DATA;
! }
!
! return INVALID_DATA;
! #endif
! if (SHORT(b1) && SHORT(b2)) {
!
return INVALID_DATA;
! }
!

! return INVALID_DATA;

}


void parseFineOffset(unsigned short scanP, unsigned char scanBit) {

+ #ifdef TIMINGS
+ unsigned char buffer[DATA_LENGTH*3];
+ #else

unsigned char buffer[DATA_LENGTH];

+ #endif

rfRetreatBit(&scanP, &scanBit); skip last bit


for(int i=DATA_LENGTH-1; i>=0; --i) {

UCHAR8 byte = 0;

+ int b = 0;

for(int j=0; j<8; ++j){

! b = fineOffsetBit(&scanP, &scanBit);

if (b == INVALID_DATA) {

return;

}

! if ((b & 1) != 0) {
! byte |= (1<<j);
! }
! if ((b & 2047) / 2 > 20) {

byte |= (1<<j);

}

+ #ifdef TIMINGS
+ if (j == 0) {
+ buffer[DATA_LENGTH+i] = (b / 256 / 16);
+ buffer[DATA_LENGTH*2+i] = (b & 2047) / 2;
+ }
+ #endif

}
buffer[i] = byte;

}
rfMessageBeginRaw();

rfMessageAddString("class", "sensor");
rfMessageAddString("protocol", "fineoffset");

+ #ifdef TIMINGS
+ rfMessageAddHexString("data", buffer, DATA_LENGTH*3);
+ #else

rfMessageAddHexString("data", buffer, DATA_LENGTH);

+ #endif

rfMessageEnd(1);

}

comment:8 Changed 6 years ago by jyrki kuoppala <jkp@…>

Duplicate removed.

Last edited 6 years ago by jyrki kuoppala <jkp@…> (previous) (diff)

comment:9 Changed 6 years ago by jyrki kuoppala <jkp@…>

Get an error from Trac, but the posting seem to go through anyway. Apparently the number signs (# number) give the errors, probably have to be quoted or avoided.

Trac detected an internal error:

ValueError: invalid literal for int() with base 10: 'assigned'

There was an internal error in Trac. It is recommended that you notify your local Trac administrator with the information needed to reproduce the issue.

To that end, you could a ticket.

The action that triggered the error was:

POST: /ticket/90

TracGuide — The Trac User and Administration Guide

Last edited 6 years ago by jyrki kuoppala <jkp@…> (previous) (diff)

comment:10 Changed 6 years ago by jyrki kuoppala <jkp@…>

can be removed

Last edited 6 years ago by jyrki kuoppala <jkp@…> (previous) (diff)

comment:11 Changed 6 years ago by jyrki kuoppala <jkp@…>

I proceeded with some modifications, debug output clarifies how the debug data (bits1 & bits2 fields) is passed by the firmware to the host.

dataclass:sensor;protocol:fineoffset;data:AF4F8FF5F57A7C7FAFAB;bits1:FAFAFAFAFAFAFAFAFAF9;bits2:4220202120100F202121;
hexdt:AF4F8FF5F57A7C7FAFAB
bits1:FAFAFAFAFAFAFAFAFAF9
bits2:4220202120100F202121

and with one more extensive option:

class:sensor;protocol:fineoffset;data:5A9F1FE7EAD4F8FF3F56;bits1:F9FAFAFAFAFAFAFAFAFAFAFAFAFAFAF9FAFAFAFA;bits2:10411020200F2010202020202010102120202020;
hexdt:5A9F1FE7EAD4F8FF3F56
bits1:F9FAFAFAFAFAFAFAFAFAFAFAFAFAFAF9FAFAFAFA
bits2:10411020200F2010202020202010102120202020
data:class:sensor;protocol:fineoffset;id:99;model:temperaturehumidity;humidity:86;temp:-1.3;publishData +R

Last edited 6 years ago by jyrki kuoppala <jkp@…> (previous) (diff)

comment:12 Changed 5 years ago by roberth andersson <sm6yvr@…>

Any news about this? Is there any custom firmware that I can use to debug new sensors?

Note: See TracTickets for help on using tickets.