Peeling the vera onion from skin to core

I stumbled into the home automation field about 10 years ago, when after purchasing a newly built house which. had a poorly located light switch in the living room. I wanted to be able to remote control it and got a GE zwave remote and a zwave light switch. Little did I know that this would lead me into looking at a vera light and wanting to do more. It was an expensive device back then… >$150. The vera light, then just transitioning to UI7 had a great GUI, fairly intuitive but had many mysterious behaviors. Treating it as more novelty than appliance, I accepted some lack of reliability while I was expanding my network. Shortly after this, was the release of the vera edge for which I was an early adopter. I bought one at full price also >$100 but did not migrate until I moved to yet another house. I thought that I could leave my vera lite behind and sell the value of all my innovative work but though all the potential buyers loved the features, the maintenance and lack of reliability scared them away. The final buyer actually begged to take them out. The high frequency of luup reloads, the random lags and frequent inexplicable crashes would scare away any buyer.

This didn’t deter me from starting again from scratch with a vera edge in the new home. This time though I started focusing on reliability and these horrendous mysterious behaviors. With an engineering background but very shallow knowledge in software I dove into the vera peeling layers after layers as I had migrated to a vera plus in the meantime to enable what one of my friends raved about: the vastly cheaper zigbee standard.

I have shared this adventure at length on this forum:

• The first layer to peel was the internet. Indeed a number of the reloads were caused by internet disruptions or communication errors either from my ISP or from the mios servers. Several of my friends on SmartThings were complaining about the same thing and I came to the realization that the first step to a stable home automation system is not only local processing but complete independence from any servers for core functionalities. The vera in spite of its early claims of cloud independence had grown very dependent on its servers with a large number tunnels and API calls which are monitored and would cause the vera to needlessly crash or reboot when disrupted. This first discovery led me to write and share a script to eliminate all of these connections… I was successful down to one very last one: The event server which accumulates alerts data in two locations and repeatedly attempts to send them to the mios server to the point of crashing the luup engine when failures accumulate. I was in close contacts with support back then and they even sent me some modified firmwares to try to alleviate this problem. No success. I am very grateful for the support and helpful attitude though and I stayed loyal. At that point I managed to decrease the reload rate from several times a day to less than once a day. I had disconnected my unit from my mios account and got rid of all the unreliable cloud features from geofencing to notifications, finding other much superior ways to achieve the same result and even more. This is when I also started migrating my scenes and plugins to openLuup.
• The second layer was my discovery of data corruption. The main source of it is the shockingly poor partitioning of the storage spaces on the vera, which negates the wear leveling and long endurance the hardware was designed for. This lead to extroot: Moving the working partition and all the vera data to a much larger SSD, allowing greater reliability and preventing premature wear and data corruption. This brought the interval between luup reloads from every couple of days to about once every 5-7 days. This was in spite of not having any plugin or scenes running on the vera while openluup was humming along.
• The third layer had me look into the logs and what a catastrophe! How could any dev and any of support guys who looked at them before think this was normal? This lead to my post about the nightly heal, unnecessary polling, poor automated wakeup configuration and superfluous neighbor node updates hogging the zwave network. It was the cause for 60-80% of random delays and about half of the luup reloads. The two layers above lead to the discussions with @edward and @melih and the release of 7.30 and then 7.31 and improving the random reload interval from 5-7days to 12-16 days. But reloads were still happening.
• Now the fourth layer, still digging in the logs I still noticed 3-4 times a day “tardy callback events” where the luup engine would hang and wait for who knows what. Almost all of them, I correlated to zwave command queue blocking jobs following “got CAN” errors or “NONCE_Get flood”. I will not go into the details but they both are results of grossly trivial cascading mistakes from bad firmware writing very unique to the vera platform.
• At one point I had even suspected that the hardware was part of the problem and had run my vera plus entire firmware on a completely non vera hardware using generic and upgraded zwave and zigbee sticks on a much more powerful x64 platform through emulation but… the results were exactly the same down to the errors and the frequency of reloads.
• The final peel was therefore the luaUPnP program, aka the luup engine itself, the core of the onion which had to come off and this was the ultimate mios program I had not taken down… It is now replaced by z-way and all of the sudden, the light took over darkness. All the zwave lag, instabilities, random communication losses… gone, everything works with full 100% reliability now for over 2 months and I have the added benefits of getting S2 support and smartstart. None of this would have been possible without @akbooer and the community in general. The migration required a  bit of work but hopefully others will benefit from the experience. The Zigbee network was also moved to HomeAssistant as even that portion, being much more stable on the vera than z-wave eventually gave in to the much better stack implementation of the ZHA component of HA.

Throughout all this adventure, I have investigated and evaluated practically every other home automation platform available from the German iobroker which I used as a side kick for some time to homeseer, hubitat, openhab, Home assistant and finally z-way and had remain loyal to the vera both because of community here and the investment I had made in my setup.

Community

What I have gotten from the vera community is priceless. From automation ideas, codes, integrations, plugins, troubleshooting and fixes, this was an awesome adventure I do not regret. It is the first forum in which I have become active and maybe had become too attached and addicted to. I tried my best to give back whatever I could so everyone could benefit from my learning and made their systems better the same way they helped improve mine. I can’t express in words how grateful I am to the likes of @rigpapa, @reneboer, @amg0, @akbooer, @blacey, @futzle and I am missing a lot of them I am sure. We saw so many of these goodbye posts from members over the years who came to a similar realization as I have now, migrating to another platform out of frustration from the immobilism from mios allowing the platform to fall behind and now ezlo introducing its new platform which I can only describe as disruptive to its community, immature, breaking everything which made the success of vera attempting to make it arguably better and allowing what broke it, break it even further. The marketing and the surreal direction the new ownership is taking, aiming at a very different market than what vera was, sadly was my last straw. I have mixed feelings of joy sadness and disappointment.

The Present And The Future

After the takeover from @melih and eZlO, a long awaited glimmer of hope came: Finally resources and a desire to move the product forward. There was excitement and life back on the forum with new firmware 7.29-7.30-7.31, all of which were forward steps in spite of some painful mishaps. There was a promise of a new firmware fixing all of the previous issues. Enough to gain my patience for another 15months. Now with the new products on the horizon, this momentum appears to have been lost and any reason I had left to remain loyal anything but gone. We have not heard of 7.32 and instead are experiencing a barrage of misleading and false marketing. Testing of very raw and not yet beta ready firmware is available and release of new products which are an updated, cost reduced, less reliable version of the vera. The specifications and the chipset used on eZLO’s products are aimed at the low end toy for the kindergartner market unsuitable for controller running a house 24/7 like the vera was.

Resolving or improving on issues like device support for z-wave and zigbee platforms, an issue which practically only the vera has became a feature of ezlo. This is a problem z-way, hubitat or home assistant do not have. They support the communication stacks. Not the specific devices. If devices are very “special or proprietary”, they allow the community to write codes to connect the stack to the UI therefore making device support practically limitless. The vera tried to automate this process, disabling this possibility and so is eZLO. I advocated for a change of strategy in vain but instead we get this pointless claim of “we support xxx number of devices and counting”. This turning a partially and unverified solved problem of the vera into a feature and selling point for the fools.
In the meantime we are not able to test the new core and verify that the vera's command queue problem does not exist with eZLO because the firmware is not fully functional, does not have a webUI and is as buggy as a can of fish bait.

The “we support everything” integration is an impossible and false promise filled with either ignorance or dishonesty, especially with the new hardware. A quick benchmark at home-assistant should be an eye opener. How could you compete with integration development and maintenance from a community of several thousand members? And they don’t even dare to make such claim and arrogant claim. To illustrate this: Does eZLO support local facial recognition from a doorbell or any ipcam and is then able unlock your Z-wave door lock completely locally? Can it connect to my car and check fuel status or remotely lock it, preemptively turn on the car AC? can it check my sleep patterns and adjust blinds and lighting accordingly? Well the ezlo plus doesn’t have the horsepower to do that. Mine does and has done so for a couple of years and yes the vera was a part of it… I did all of this with openLuup coordinated with homeassistant on much more powerful hardware with the vera as a simple device bridge. Capabilities far beyond the reach the ezlo plus hardware. So can the ezlo replace the vera as a simple device bridge? Not really, the API is different, the most basic triggers do not work, we can't even really test anything since it is mobile app focused.
And what to say about the move to eMMC storage? One largest complain of all these systems running on the rPi? It’s the failure and data corruption of the controller less SD cards which is exactly what the eMMC is except that you cannot replace it. Ezlo products are toy grade… A downgrade in quality and reliability from what the vera was with a number of minor updates, currently of little use.
The Ezlo product lineup seems to be aimed at the naive entry level shopper at walmart looking for a cheap alternative to a Philips hue setup and I bet will one day find its place next to the iHome speakers in the surplus stores like Ross or Marshalls. I can’t trust my home automation to such a low grade piece of hardware even if the firmware was perfect and it is not looking like it will be for a long time if ever. It isn’t even functionally equivalent on the paper roadmap.

The lack of a migration path for current installations: scenes and plugins makes it even less functional equivalent and I believe this is a massive strategic mistake as no existing customer can ever really migrate to eZLO with vera as a base and will therefore be much more likely to migrate to another more mature powerful and/or feature complete platform. All these criticisms on the forum were meant to be constructive and help the management to a better understanding of the market and a product but they have all taken with extreme defensiveness to the point of denial and mostly ignored.

So I will take the test reports from my happy system as well as myself off of the vera forum and will go contribute elsewhere. In a place where I am not told what to say or what to think as this is what eZLO has turned this forum into and where I can continue to learn, grow and contribute all with a positive and joyful spirit. 

Rafale77