MILESTONES.md

Milestones

Milestone Eleven- m11

1. TADSummit Interviews

   In the Erlang-Red walkthrough, I created examples flows for implementing diameter protocol. I also did an interview talking about the ideas and motivations behind Erlang-Red.

2. Process Node and Process tree support

   Initial support for introspecting existing Erlang processes is now provided by the process node that can be used to represent an existing Erlang process. The process node can be integrated into existing flows and be sent messages just as all other nodes. Entire process trees can be imported using the process sidebar.

3. Capture/IO node

   Used in combination with the module node, a captureIO node captures any IO generated by the module code. The captured IO can then be passed to the debug panel.

4. Proper support for credential files

   The flow_creds.json file stores configuration node credential details separately from the main flows.josn. At the moment, the file isn't encrypted but this will change in future releases.

5. RabbitMQ/AMQP client node and Kafka nodes

   Initial support for AMQP and Kafka consumer and Kafka producer.

Milestone Ten - m10

1. No new nodes - consolidation milestone

   No nodes have been added in this milestone. I have been concentrating on consolidating the codebase and refactoring. That does not mean that nodes are complete, rather that bugs haven't been fixed.

2. Screencasts

   I was asked to participate in a discussion over at the TAD Summit. I talked about why I am doing this and what my aims are, hint: they have nothing to do with Erlang!

   That chat motivated me to create screencasts to better present what Erlang-Red can and cannot do. It meant that I fixed bugs and made improvements to the flow editor user experience.

3. JSONata bumped from 0.4.4 to 0.5.0

   Erlangs JSONata implementation now understands the $not and $match operators which were required for something - cannot remember for what. Fixed a hash map thing that values should not be lists but binaries. The conflict between atoms, lists and binaries will continue. I have decided on binaries as being the string representation within Erlang-Red.

   This reminds me of improper_lists and JSON not encoding these. For me, improper lists aren't lists, howeverthey reply to is_list/1 returns true for an improper list. Improper lists then promptly break each and every single list function. I just don't understand Erlangs thinking here. Thankfully there is a solution using a guard since there is not an is_improper_list/1 function.

4. Single user, no more multi-user support

   I decided to partially remove multi-user support by notifying users of changes made to flows. This is because I wanted to have a single flows.json file on the server. For that, multiple users is not good. Most of the functionality is in the runtime manager, so that it can be reverted to a multiple user environment. On the other hand, this brings Erlang-Red in line with Node-RED in having one-user per server, one flow execution per server.

   Turns out though, most of the multi-user support will remain since the web-socket identification of users will continue. Hence there is a strange non-appearance of debug messages in all browsers unless a user also deploys a flow - the execution of flows will remain per-user basis.

5. Flow editor with a new coat of paint

   To prevent confusion, I did change the banner colour to a bluish-off-black and removed the logo pending a new logo for Erlang-Red. I did consider using the eye of HAL as the logo but that is probably copyrighted by the estate of Stanley Kubrick and/or Arthur C. Clarke.

   At least good to know that HAL uses Erlang-Red. Perhaps Chuck Norris will also give it a try.

---

Milestone Nine - m9

1. New Nodes: batch, range and filter

   Batch node is used to mark messages as belonging to a batch and buffer messages until batches are completed. These messages can later be bunched together using a join node. Basic example shows how this can be done.

   Range node is used to map between two different value ranges, mapping an input value from an input range to an output range and creating an output value based on the output range. Test flow demonstrates this functionality.

   Filter node filters messages based on changes in value. Can be used to block messages until a specific value changes. Test flow demonstrates this functionality.

   Each node is documented within the flow editor, start there for more details on these nodes. Also nodes are compatible to the Node-RED counterparts, so documentation related to the Node-RED nodes are applicable to the Erlang-Red nodes.

2. Upgraded JSONata parser to 0.4.4

   Added several new functions including base64 encoding and decoding. The more functionality in JSONata the Erlang code needs to be defined.

   JSONata is essential to Node-RED and hence Erlang-Red and good knowledge of JSONata can save much time when creating flows. JSONata aims to replace simple and repetitive functionality normally done using a function node. JSONata can be used in the change, switch and inject nodes and thus prevents the usage of a function node and hence make for smaller and compactor flows.

3. Two hundred and fifty compatibility an regression tests

   To ensure compatibility with the original Node-RED functionality, I have been creating visual flow test suite executable both in Node-RED and Erlang-Red. These compatibility tests are located in a separate repository for others to also use. For example, for creating a Node-RED based on rust.

   All tests are integrated and maintained here and now have reached a grand total of 252 - at time of writing. All tests can be escaped within Erlang-Red using the unit testing package - which has a sidebar panel for triggering all or individual tests.

4. Added GitHub actions for performing unit tests on update

   All compatibility tests can also be run using eUnit and configuring an GitHub action to execute these on push makes good sense.

5. Using ETS and pre-compile JSONata for better performance

   With the help of @vkatsuba, I started to use ETS for storing messages more efficiently. This was particularly useful for the batch node that potential needed to store many thousands of messages.

   In addition to that, I began to pre-compile JSONata stanzas to make JSONata usage faster and also better since errors are identified at deploy time not run time.

---

Milestone Eight - m8

1. Moved type parsers out to own repository

   With commit, moved type parsers (for number, binary/buffer fields and attribute access) to a new repository for reuse in other projects and reducing the code base size.

   Also defines the responsibilities of the parsers and the node code.

2. New Nodes: Sort, Binary and I2C

   Node-RED Binary node implements a Packet definition language to make it simpler and universal. This node is now part of Erlang-Red and was used to implement PNG image parsing.

   Using a more universal binary definition, especially one that is defined in Node-RED, makes flow code more portable and neutral than using Erlang specific binary matching. See the forum discussion for more details. Perhaps other meta-description specifications use kaitai would have been better however the point has been made.

   The Packet parser implementation is outsourced to the parser repository and was extended to including reference of values defined in matching the binary format. This allows for chunks to define lengths and these chunks to be read in one cycle.

   Created an initial sort node implementation including a number of unit test flows to demonstrate the nodes features: sort by parts.index and sorting array of objects.

   Using the circuits_i2c Elixir library to implement a basic set of i2c nodes. This was then used to interact with a Scroll pHAT LED PCB. This was a flow was inspired by the same functionality for Node-RED.

3. Embedding on Raspberry Pi

   Erlang-Red embedded on a Raspberry Pi 4, admittedly using docker and the docker hub image but it worked. The goal was to control the Scroll pHAT via the I2C bus using the I2C node.

4. Diverging from Node-RED flow editor

   Along with modifying the defaults for the switch and debug nodes, I now added base64 decoding to Template node. This makes using binary data in flow tests simpler. Binary data can be encoded in base64 and then decoded by the template node before being passed through flows.

   Another minor modification was allowing underscores in the definition of hex and octal and binary values. This was inspired by the reading the specs of kaitai format descriptions. A good source of various binary formats.

5. License Smorgasbord

   Using a trifecta of licenses so that everyone can pick and choose which license they wish to use. Of course, this is comment on the fact that open source software has been a battle of software license and which project can use the most permissive license.

   Its a pity that GitHub does not have a indication of innovation or novelty when rating codebases on their site. Instead the first factors are documentation, licenses and activity:

   

   As if activity is an indication of quality, it's an incentive to add more features and bugs so that projects need constant maintenance to keep working. Using the activity as a measure of project quality ensures the opposite and also ensures that projects are never completed.

---

Milestone Seven - m7

Code Name: The Non-Plural Milestone.

1. Docker image @ hub.docker.com

   Started using proper release numbers and with that, decided to create a docker image that is available at hub.docker.com. That makes it simpler to try out Erlang-Red using a single command:

   docker run --tty --publish 8080:8080 gorenje/erlang-red:0.2.2

   That will start the web-server on localhost:8080 and also drop the container into a Erlang shell. Thanks to @joaohf, the size of the container was much reduced, now down to 21 MB compressed.

2. New Nodes: ClientCode, Tcp Request and File

   ClientCode node is a hybrid frontend-backend node, it should definitely not be used in production but for certain development purposes it can be a fun node to use. The node is part of the introspection package and provides a structured method to trigger fronted actions (in the browser) via backend flow events. Example usage is drag & drop activation.

   Tcp request node is the last of the tcp node collection and is a bidirectional node to be able to send and receive tcp packets. File node is the counterpart to the file-in node, the file node is used to write and delete files to disk.

3. Http-In multipart support

   Initial support for file-based HTTP multipart content. This was amazingly simple is Cowboy is so amazingly simple - once you understand it! Cowboy is really nicely designed, even if a redirect isn't builtin - or at least, I didn't find it.

4. Code cleanup and refactoring and Macros

   Began going through my naive Erlang code and making it less-understandable by using macros and function argument matching. Makes everything compactor but also, sometimes, less understandable. I can already smell the second wave of refactoring rolling on over the shore.

   I really do have a love-hate relationship with Macros - ?IWonderWhy. I defined a bunch began using them and then, a few days later, I regretted it. But I'm sure that has something to do with the weather. If I get the naming right and get the right mix between functions definitions and macros, it probably work out.

   I have also been told not to use bang ! because that's not a done thing in Erlang. I actually use it for internal-internal things, such as the tcp router which maintains open ports and shares data amongst listeners. Personally I don't see where the issue should be in using a construct that defines the language. In certain sense it's similar to saying I'm not allowed to use the letter 'a' for variable names.

   -spec register_connector(
       HostNameOrIp,
       PortNum,
       PidTcpInNode
   ) -> Result when
       HostNameOrIp :: binary(),
       PortNum :: 0..65535,
       PidTcpInNode :: pid() | atom(),
       Result :: {connected, binary()} | connecting.
   

   Btw why is that when and not where? If I define the types of the parameters, then surely it's where the hostname or ip is of this type and not when.

   We're all inconsistent but not when it comes to criticising others inconsistencies.

5. Title change - Erlang-RED becomes Erlang-Red

   I decided that capitalising on red isn't really for me, in fact, my ears got tired of all that shouting. So I decided to use some more of the other half of the ASCII standard.

   But as Shakespeare said: "What is in a name? Does not a rose by any other name still smell as sweet?".

   Speaking of inconsistencies, I decided to drop the 'v' prefix in version numbers, instead just using the version as tag. This is of course is inconsistent with the 'm' prefix in milestone tags. I didn't see a consensus in the usage of 'v' or not, so I decided that a rebar config looks better with the tag name does not contain a 'v'.

---

Milestones Six - m6

Code Name: Milestone Six.

1. New Nodes: Tcp/IP, Event handler, Generic Server

   The main Erlang behaviours are now represented, gen_server and gen_event have now been included along with gen_statem and supervisor. All nodes are compatible with the supervisor node so restarts/stops can be modelled and implemented. Basically it is now possible to model complex Erlang architectures.

   Tcp nodes have also made it. These are implemented as close as possible to the original nodes defined in Node-RED. Their tests are still "in progress" but with the creation of the MQTT broker flow, the tcp nodes have a good testbed.

2. Breadboard Programming & AI ignorance

   The concept of using breadboards as a prototyping technology in software development is an innovative idea that warrants exploration.

   AI thinks this is a good idea so I went ahead and began using this terminology. I created two examples of what I mean by breadboard programming: modelling a single tcp connection and the MQTT broker from above.

   

   For me having another analogy of what I am doing here is very helpful. It provides a little bit of a focus compared to thinking of this project as being a "visual flow based programming environment for Erlang heavily influenced by Node-RED". I dunno, breadboard programming seems to be more marketable.

   Speaking of marketing, I also change the license of the project to make it more "FAANG" compatible. Immediately I was shoot down by other wisdom. F2k It: it's GPL all the way down, along with the turtles.

   Speaking of f2k it, I added an .aiignore file to indicate which artificially intelligent contributions are willingly accepted by the project. I didn't ask AI about this idea, I just found it good. Perhaps the idea will catch on for a way to identify "stuff" that is definitely human so that humans can train AIs to be better humans.

3. Better.['support'].for.["property"].access."types"

   Speaking of better, Node-RED has a wickedly complex possibilities of accessing properties on the message object. I noticed this when implementing the MQTT broker from an existing Node-RED flow. The author used access methods that were new to me. So I went down the rabbit hole ....

   I ended up creating a parser for all the various possibilities and the flow test hopefully covers most cases ... although then there are some edge-cases that I haven't yet implemented.

   The weirdness comes in when considering empty message objects on which values should be set. For example msg.key1[2].key3 = "fubar" will create this structure automagically:

   msg = { key1: [undefined, undefined, { key3: "fubar" } ] }
   

   So that key1 becomes an array which contains an object at index 2 and undefined at index 0 and 1. This Node-RED does automagically because Javascript mixes accessing methods for hash and arrays. The '2' can be interpreted as a hash key or the number 2 as array index. And so when used as a string, as in msg.key1['2'].key3 = "fubar", it becomes a hash key:

   msg = { key1: { '2': { key3: "fubar" } } } }
   

   And that makes all the fun and games of implementing this stuff. Either way, I managed to get things working mostly, aiming to be 100% compatible to what Node-RED does. Also: single quotes and double quotes are the same in Javascript and both represent a string.

   In Erlang-Red I modified that semantic to be that if a value is placed in single quotes, then it becomes and atom, so that msg.key1['atom'].key3 = "fubar" becomes:

   msg = #{ <<"key1">> => #{ atom => #{ <<"key3">> => <<"fubar">> } } }
   

   So 'atom' becomes atom and string keys become binaries. I decided to add atom support for accessing data that might come from an external data source or whatever. These property specifications can be used to access data as-well as setting data on the message object/map.

   As part of a previous milestone, I decided to change all map keys to binaries instead of a mix of atoms and binaries. It was a good decision to unify the key type - both because of atom limitations and consistency. That decision made implementing this much simpler.

   I do however use atoms for internal keys and that is good, since then its clear which keys are externally usable and which keys should be seen (i.e. debug log) but not touched. Most of the internal atoms also use an underscore prefix - another indication of their internalness.

4. Change, switch and inject nodes now support binary and buffer values

   The number field support in Node-RED is also very generous. Many forms of floating point numbers are supported and also binary and hexadecimal, both positive and negative.

   Turns out that it is so complex, that I had to build another parser just for handling the number field. All that parser does is transform Node-RED floating point representation to a representation understandable to Erlang. The test flow attempts to cover many of those cases.

   As a consequence of creating the parser, I also began to support binary 0b1010101110110111100 values and hexadecimal 0xfeedd06 in the num field which is also a Node-RED thing. This then also lead to supporting the buffer field type which is nothing else than JSON.parse([<<json arrray definitiona>>]).map( d => parseNumber(d) ) in pseudo code where parseNumber does the same as a num field above.

5. Breaking with Node-RED compatibility

   Having spoken of being compatible with Node-RED and aiming for magical 100% compatibility with Node-RED, I have decided that its never good to be 100%. Being the leader you lose motivation and continually look around you, worried about being overtaken. Mainstream is best, middle of the road, no need to succeed.

   And therefore I made a couple of changes to the core node defaults because they have always annoyed me - 1) the debug node dumps the entire message to the debug log instead of msg.payload as is default in Node-RED. That is because I implemented the dumping of specific message attributes to the debug log and now the Erlang-Red debug node would also inflict this default setting on me. Why do I prefer to dump the entire message? Because mostly my problems don't lie in the payload...

   Secondly the switch node has a "check all rules" by default setting. Imagine a case statement without breaks (not even but similar). Which I also dislike. I continually alter that default setting every single time I pull in a switch node in. So that the default in Erlang-Red is now "stop after first match" and if you want "check all rules" then you'll have to change it.

   My issue with "check all rules" is that it is extremely rare that you would want to be using that - perhaps this is an IoT thing or something that one does in industry - but me, I and myself, have - honestly - never needed to use that setting.

   Perhaps its also naming: switch for me is a switch statement, filter is a something else. This "check all values" makes the switch node into a filter node. Which reminds me: pity node aliases aren't supported - nice feature to be able to define a filter node that is a switch node but with a different setting. So that your personal settings can be defined as palette "nodes" but are actually just existing nodes.

   So anyway, F2k It bye-bye 100% Node-RED compatibility, hello freedom of authorship and experimentalism.

---

Milestones Five - m5

1. Erlang-Red Message Tracing

   Using the introspection node package, implemented message tracing of Erlang-Red messages.

   

   Note: that there are two types of messages in Erlang-Red - the Erlang type of messages and the Erlang-Red types of messages. A disadvantage of using a message passing based programming language to implement a message passing programming paradigm like flow-based programming! Either way, this feature traces Erlang-Red messages (the kind passed between nodes) and not Erlang messages (the kind passed between processes).

   A second point - related to the feature - message tracing can be activated at any time and does not require deploying the flow to the server, i.e., message of a live running server are being traced.

   A third point is that this feature was originally implemented for Node-RED and all I did was port the backend to Erlang. Now I have the same insights that I have in Node-RED but in Erlang-Red.

   One more point: this feature can definitely flood the flow editor into submission. Use with care. The server should be fine since all it is doing is sending messages out on a web-socket via an exclusive process for the web-socket connection.

   A fifth point: in the message tracing, the status of the node is abused to have a "msg received" status appear. That message (only in Erlang-Red) now contains the process id of the underlying Erlang process for the node.

2. Nodes for gen_event & gen_statem behaviours

   Erlang-Red has become more Erlang-like by not only implementing the supervisor behaviour but also the gen_event and gen_statem behaviours. Both of these make use of the module node. The module node can be used to implement complete Erlang modules in Erlang-Red. These modules can then be used to define the behaviour of the gen_event and gen_statem nodes.

   The event handler is completely dynamic with modules being added as handlers at runtime, the state machine is static with a module being added at design time. A state machine handler can however dynamically replace the handler if so desired.

   The event handler will need some more TLC to make it static and dynamic. The overhead of configuring the event handler at runtime is strictly speaking not necessary. The corresponding test cases: eventhandler and state machine.

   When I started out on this project (all of 2.5 months ago!), I intended to implement Node-RED features and nodes in Erlang. Now having created the Erlang-only nodes which implement these behaviours, I'm thinking that wouldn't it be nice to create Node-RED equivalent nodes. Half the work is done: these nodes are all packaged into an Node-RED node package and can be installed into Node-RED, only there they do nothing. However having a supervisor to ensure that things get restarted in NodeJS can be, perhaps, just maybe, also useful.

   The assert nodes that I created actually do work with Node-RED and Erlang-Red, so these are the first ever nodes that are cross platform.

   The deeper philosophical point I'm trying to make here is that a visual flow-based programming approach can lead to cross pollination of programming ideas. It can also lead to interesting learnings around programming paradigms. But for those sceptical about visual programming and the benefits of it - especially how can a "serious" application be developed visually in a browser - go back to using punch cards. We came from giant mechanical calculators, went to punch cards then to keyboards and now we're stuck at Artificial Intelligence and typing in prompts to get code we might or might quite understand. Amazing, awesome, I'm so excited - truly. As programmers, we haven't started to use the true benefits of visually constructing our programs and applications, instead we claim that the keyboard is the only instrument for creating instructions for a piece of metal with many wires and many CPUs. Somehow it does remind me of knocking two stones together to make fire and the amazement when someone comes along with an Artificial Stone in the form of lighter.

3. Improved documentation using the Flow2UML node

   I dislike writing technical documentation. I consider code enough documentation - make the code read like a good murder mystery and presto it's a clear whodunnit. But Erlang code is damn compact and even I don't understand why and what I did two months ago. Not good. The second thing I dislike is creating flow diagrams - either visually or textually. But there is no way around it. If I want to be able to understand how things work in six months time, I have to create documentation.

   Mermaid seems to be the tool of choice for such things so back in the good days of Node-RED, I created a node package for converting flows to Mermaid-UML-like diagrams, i.e. a flowchart. That flowchart, being defined in Mermaid syntax, can be embedded and displayed in Node-RED flow documentation - neat! But because I wanted extra features, I created the mermaid-flowchart node for creating more complex flowcharts.

   Some examples are here, here and here. And inside Erlang-Red:

   

   But creation is one thing, usage is another - how I do use these diagrams? Well Node-RED supports embedded mermaid diagrams and by extension, so does Erlang-Red:

   

   That's the nice thing, the documentation of a flow is embedded in the flow and goes wherever the flow goes. I created red-erik.org to host that documentation in a webpage. So for example, the gif above is online at red-erik.org.

   For any Node-RED readers: the original flow2UML package also contains the new mermaid flowchart node. So it's possible to create these complex flowcharts in Node-RED also.

4. Timeout support for function and link-call nodes

   This might seem minor, but both required a complete re-think of the Erlang architecture. The function node now creates up to two Erlang processes per Erlang-Red message received. The Erlang architecture for the function node is documented and describes why this needs to be.

   Timeout for the link-call node was much simpler since it can be implemented using the Erlang-Red message object. Basically an erlang:start_timer is set when the Erlang-Red message is passed onto the link-in node. If the message comes back before the timer is triggered, the timer is cancelled. If on the other hand, the message isn't received, the timer is triggered and a timeout error is raised. In creating the unit test for this feature, I realised that the message - once it arrived - is still passed on! That is Node-RED behaviour, so Erlang-RED copied that. IMHO that doesn't make sense since the message timed-out, however I can understand that if the message makes it back, then why lose the data - obviously took a long time to get it, so don't waste it!

5. Better documentation describing the Erlang architecture

   I am starting to be happy with the Erlang architecture so I've started to document it. Starting with the supervisor node, a technical description of how it interacts with the Erlang code base to implement Erlang-Red node supervision using Erlang process supervision. There is technical description of how the link call node works.

   Once I'm through with documenting everything, I will probably end up modifying everything again - programming is truly a Sisyphusian activity.

---

Milestone Four - m4

1. Supervisor Node

   One of the most important concepts of Erlang is the supervisor behaviour. It defines an elegant approach for monitoring, restarting and stopping processes that are unhealthy. It thus allows for complex yet stable architectures to be created in Erlang. Erlang-Red now has its very own supervisor node to provide the same behaviour but visually.

   The supervisor node is fully feature compatible to the supervisor behaviour and the node provides a simple mechanism of selecting nodes to be supervised and also ordered. Supervisors have a specific ordering of processes and this is also supported by the supervisor node by being able to sort nodes, each node being a process.

2. Added Pencil to Flow-editor

   

   Might seem silly but it is important for documentation. Test flows can now be viewed over at Red-Erik.org (hosted via flowhub.org), so that test flows can now be better documented via "pink link technology", for example, flow one, flow two and flow three.

   Pink links highlight nodes and groups in the flow diagram making it simpler to explain flows within the flow description. These pink links are a html stanzas, e.g.: <a class='ahl-group-only' data-ids='0bf41347dde34db0'>name</a> - which are a real pain to write by hand. Node ids had to be copied, the class name typed out, node ids found from the flow editor ... blah blah :(

   Also important to note is that flow documentations is found via the flow tab:

   

   This means that flow documentation lives with the flow code and is displayed both in Erlang-Red (within the flow editor) and at flows.red-erik.org. (All acknowledgement to the Node-RED team for putting this in the editor, I am just piggy-backing their work and extending it to my needs.)

   Back to the pencil: I now select a node or group from the info panel and click on the pencil and I have my HTML stanza:

   

   The nice thing is that the info panel is usable while editing the documentation for a flow, so the pencil can always be used to create the HTML stanza for the pink links.

3. FlowHubPull node: dynamic flow code loading

   FlowHub.org is my attempt at creating a visual code hosting platform. Part of that is defining interdependencies between flows. For me a flow is the flow tab (for others a "Flow" are all flow tabs). I like to think in flow tabs and creating functionality that fits into a single flow tab.

   Flows can be linked via the link nodes and I do this very often. But what is missing is the loading of flows into Erlang-Red dynamically. So for example, this test flow first triggers the flow hub pull node to load another flow into Erlang-Red. It then triggers the test which contains a link to the other flow.

   The FlowHub pull node is currently limited to the flows contained within the Erlang-Red test-suite but there isn't any reason - other than laziness - why this cannot be extended to include other sources of flow code.

4. Remove catch around function node

   With the introduction of a supervisor node, I removed the exception handling around the function node - I actually had to do this to get the function node to fail (was using a divide by zero error to test the supervisor node).

   But it made me realise that Erlang is a lot different to NodeJS when it comes to exception handling. Erlang processes are lightweight and designed to fail, so let them fail becomes the intention. NodeJS on the other hand wants to handle all exceptions and not let the system get unstable, so there is much use of the try-catch-final pattern within NodeJS code.

   Erlang heals systems by restarting processes if they fail. It is assumed the system is stable until something fails. Perhaps the complete inverse of what NodeJS does.

   By removing the catch block on the function node, I'm moving away from the NodeJS way of doing things and moving towards a more Erlang-like approach.

   Ironically the problem with catching errors in something as generic as the function node (which takes straight up Erlang code and executes the code) is that: how to deal with the exceptions? All I was doing was pushing it to the debug panel ... but I could do that far more descriptively by using a catch node connected to a debug node.

   Its only a minor change yet it represents a major step forward for Erlang-Red because it allows Erlang folks to think in terms of Erlang when using Erlang-Red.

   Additionally Erlang-Red now has two nodes for error handling: catch node for catching exceptions and the supervisor node for restarting failing processes. As they say in German: twice holds better!

5. Erlang, Elixir, BEAM?

   Tried but failed to get Erlang-Red compiling in an Elixir environment. It does so but then Erlang-Red won't compile in an Erlang environment. I decided to focus on the Erlang path - after all its Erlang-Red not BEAM-Red. My experience is not up to getting Erlang-Red to compile for both Elixir and Erlang, perhaps someone else will make in-roads there.

---

Milestone Three - m3

1. Elixir code constructively includable.

   Since Elixir also runs on the BEAM VM and there are some libraries that aren't available in Erlang, I made an effort to integrate third party Elixir libraries - in a structured manner - into the codebase of Erlang-RED. Many thanks to @filmor for the tip of using their exerl plugin. Works like a charm!

   This effort lead to two new nodes using Elixir codebases: the markdown node uses earmark and the csv node uses nimble_csv. Nimble might be removed again since there are pure Erlang CSV libraries - we'll see.

2. Erlang JSONata has moved out

   Moved the codebase for the JSONata parser out into its own repository. I wasn't actively extending it and moving it out will allow others to extend and add to the JSONata functionality supported in Erlang-RED. It's always sad when the children move out but its for the best!

3. Function node in Erlang!

   One of the more important nodes in Node-RED is the function node because it allows for creating Javascript code, i.e., dropping down to high-code instead of low-code. This can be helpful for adding functionality to a flow that is not covered by any available node. What the function node offers is syntax highlighting and error highlighting in code. This makes it a mini editor inside of Node-RED.

   An initial emulation of this this for Erlang has been made so that Erlang-RED flows can now also speak Erlang. The implementation is still rudimentary but it shows what needs to be done to get this happening. A function provides much power and the Erlang one is no different - i.e. it's a window into the server hosting Erlang-RED. So a sandbox needs to be created and much thought into how to ensure that nothing breaks or gets maliciously damaged.

   Use at own risk! is the conclusion.

4. Add FlowHub.org nodes for managing flow test cases

   FlowHub.org is my attempt to make visual programming truly visual. One of the first things I did was to create a visualisation of Node-RED outside of Node-RED, i.e., flows displayed in webpages. From that initial effort, FlowHub.org was created to support visual version controlling of flow code - both inside of Node-RED/Erlang-RED and also externally via a website.

   Adding FlowHub to Erlang-RED now allows me to maintain the test flow repository and easily import tests into either Node-RED or Erlang-RED - with the same consistent interface. As an aside: FlowHub is also coded in Node-RED, i.e., dog-fooding all the way down to the turtles.

5. Red-Erik.org utilises multiple flows

   The initial release of Red-Erik was based on a single, now its a multiple flow monster! Why is this important? Because it ensures that the link nodes work across multiple flows - this is their main purposes. Also it made me think about how I could get Erlang-RED to execute multiple flows. In doing so, it became clear that executing and designing flows are very much different activities. When the flows get executed, they basically just become an Erlang architecture of processes, nothing remains of the original flows. Each process only knows where to send its messages to - when it receives a message. There is no overall structure of the flow in memory. This is great because there is no overhead to maintain such a structure.

---

Milestone Two - m2

1. JSONata parser and evaluator

   JSONata is a transformation language for JSON objects that is heavily used in Node-RED. Within Node-RED it provides a vital function by offering functionality for manipulating the msg object (i.e., the data flow) without having to code Javascript code. Because of this and also wanting to be 100% compatible with NodeRED, I created a JSONata parser in Erlang and implemented basic functionality that JSONata provides. The parser is defined in yecc and can be easily extended in either the evaluator or in the yecc definition. Having JSONata support is a major step forward for the Erlang-RED!

2. External Connectivity: network connectivity via HTTP-in and MQTT nodes, command connectivity via the exec node

   Having spent the initial part of the project developing the routing and flow control nodes, this milestone has three new nodes for accessing the external world.

   The HTTP in is my personal favourite because with it, it is possible to do static http routing within Erlang-RED. Even better, combined with the dynamic linking of the link nodes, it is even possible to do dynamic web routing.

   MQTT nodes provide message bus connectivity. The mqtt nodes are particularly useful for connectivity between Erlang-RED and NodeRED: it is now trivial to have instances of Erlang-RED communicate with NodeRED.

   Exec node can be used to shell-out to a new process. Finally it is possible to read cat /etc/passwd using Erlang-RED - have hours of fun and games with family and friends by hacking each other installation of Erlang-RED ;)

   With these three nodes done, it is possible to create applications for routing MQTT traffic or HTTP traffic to MQTT or vice versa.

3. 100+ Visual test flows for ensuring compatibility

   There are just over 100 visual flow tests to ensure that Erlang-RED nodes are compatible with the current Node-RED functionality. These "test" flows can be utilised by other projects (e.g. Py-RED) to also ensure compatibility or by NodeRED itself for regression testing.

   Creating these flows has also given me insights into the specifics of existing nodes. It also lead to pull requests for the original NodeRED.

   In the long term, I will push these tests to a separate repository so that they are independent of the project.

4. Internal architecture for utilising third party Erlang libraries

   With the implementation of mqtt nodes using the emqtt library it became clear that each library will need a manager for handling communication between the node and the library.

   This pattern was also implemented for the exec node that utilises the erlexec library and which has a manager to coordinate the message passing between erlexec and the exec node.

   What I learnt was that libraries generate messages and if these are sent directly to the node, then the node behaviour would become chaos. I am torn between having generic messages and passing them off the individual nodes and being strict on the set of message that a node can receive.

   But having a manager is actually the tie-breaker, being able to remove library specific code to the manger while the node concentrates on "node specific" functionality is optimal. Done right, it should be possible to replace third-party libraries without modification to the node code. What is node specific functionality? Something like ensure that its status is displayed correctly - in this case the messages waiting at a delay node.

   Division of responsibility is simple when nodes worry about their status and managers about external resources.

5. https://red-erik.org the very first Erlang-RED application

   Red-Erik.org represents an initial headless installation of Erlang-RED. The flow being executed consists of a http-in node listening to the '/' path, two template nodes the contain the contents of the page and a http-response node that sends back the content when requested.

   

   It is oxymoronic to speak of a "flow executing" since what actually happens is that the flow represents a blueprint for an Erlang architecture. When the docker image spins up, the Erlang processes are created (four - one the http in, two for two template nodes and one for the http response node) and then messages are sent.

   The processes aren't wired together, all that happens is that the http-in node will send a message when a request from a browser is received. That message goes to the first template node that sends it on to the second template node after modifying the message to contain its payload. Using a mustache template, the second template inserts the payload into a html layout. That is then sent to the http respond node that is connected to the second template. The http respond node sends the response to the client.

   So the flow specification is exactly that: a specification for a bunch Erlang processes that are completely independent of one another.

---

Milestone One - m1

1. Working integration of Erlang and Node-RED flow editor

   Static flow editor frontend codebase communicates with Erlang via http endpoints. Flow editor codebase is retrieved from a running Node-RED instance using a well defined script. A clear path to updating the flow editor is thus assured: updates to the live instance are reflected in the static version in the project.

2. Erlang codebase

   The basic shape of an Erlang project has been followed. Initially project codebase was unstructured but many at the Erlang forum helped to improve that structure and form it in to proper Erlang codebase. Using rebar3 for release and build management, eunit for testing and ered_ as project prefix.

3. Erlang Architecture stabilised - Supervisors and Processes all the way down.

   Main gen_server components are placed under supervision to ensure stability. Nodes are gen_server processes but are not placed under supervision be will be in the long term. The ideas behind the architecture have been documented.

4. Initial collection of implemented nodes

   An initial collection of nodes have been implemented, they are not feature complete but provide the basis for initial testing. The intention is to create as many nodes as possible to ensure that the underlying architecture supports these nodes. Feature completeness will hopefully not affect the architecture in the same way as initial creation. Using the testing panel within the flow editor, it is possible to check what works and what does not.

5. Flow Driven Development Process

   A collection of approximately sixty flow tests have be created to support development. The idea is to develop these flows in conjunction with a working Node-RED installation and then transport these flows here. Erlang-RED Nodes are then developed until the functionality is implemented, that is, until the test passes. Test flows can also be marked as pending (using the flow environment so that future functionality can be staked out. This development process has also be documented.

---

Milestones are spots in the life of the codebase where everything comes together and there is stability. Milestones are not planned, they are discovered once reached. Milestones have no relation to versions.