Web Portal Notes!

I have now added the ability to view, create, modify and delete notes from the web portal. I’m using an HTML editor called “TinyMCE”, which works well and integrated fairly easily. It can’t embed pictures or media but It’s nice to have basic formatting, lists printing, etc. Because it saves as HTML code (which is good for compatibility), Kootnet Control Center’s note section does not support it, so viewing in Control Center will show HTML code instead of an HTML page. I’m not completely done the Notes section yet, as I still have to add the custom Date & Time stamp the user can edit to show wherein the database the note is referring to and I wanted to get the spell checker working.

On a personal note, the wife has left the province for work and has stints of 3 to 5 weeks at a time away from home before coming back for 1 or 2 weeks. So I have a little less time for programming projects.

The weather has been mostly cloudy grey and the temperature is kinda random from like +7c to -11c with wind, rain, snow, hail, etc. I’m looking forward to warmer and sunnier weather.

Back to my day job for now. I have a few work projects on the go too, like finishing an Exchange migration to Exchange 2019, replace some failed drives in a server and get an IP camera system working from outside the local network but through the VPN for security.

New Sensors Added & Standardized Configurations Created

I added 3 new Pimironi sensors. The MCP9600 (Temperature), MSA301 (Accelerometer) and the SGP30 (Gas Total VOCs). I don’t actually have any, so I can’t test them, but its all set for when I do, or if someone else does.

I have also decided to standardize my configuration files through a base inherited class. With this, the majority of functions used among the configurations are the same. So you load and save all the configurations the same way for instance. It has also cut down on the number of lines by a decent amount. I still have a few configurations to update, but I have already converted the primary configuration and the installed sensors. I’m working on Trigger variance configuration, then it’s just the Online Service Configurations and Sensor Control’s.

That’s it for tonight’s updates.

Bit of a break

I took a bit of a break from my main programming projects and helped make an installer for another open-source project Darkstar. I started with a deb installer but realized a script would do a better job, especially for customizations. So that took about 3 days to get working well. Doing it even allowed me to fork the project on GitHub & create my first Pull request for someone else. So that was nice to be able to contribute. The reason I choose this project was that I was requested to set up a server for my brother, so he could more easily work on his FFXI graphics improvements. Basically, he has been making his own custom textures for the game that looks really beautiful! Unfortunately, the install instructions at that time were out of date and consist of a bunch of terminal commands and a few scripts. Without the new installer script, it took a while to get it to work at all because it was missing dependencies and was written for a much older Ubuntu install 14.04. At the time I could only get it working on an Intel system with Ubuntu. Now with the interactive installer script I made, it can be installed on the latest versions of Debian, Ubuntu and Raspbian. Being able to take advantage of the cheap Raspberry Pis is definitely a bonus. With a single terminal command, one can get set up with a Darkstar server and only need to enter a user, password and IP address. So I’m pretty happy about that.

In other news, it has been snowing A LOT here. I even had to climb my roof and remove the snow there so it wouldn’t possibly collapse. The snow on my shed roof (which I don’t bother to clear), is about 3 or 4 feet tall, and the roads are basically bumpily packed ice sheets. There’s also a huge fancy snow hill at our local no-frills parking lot that’s probably 12 or more feet tall and covers a very large amount of the parking lot… maybe … 2500 Sq ft. Anywho, it’s been a very snowy winter this month so far and although I’m not super impressed, my daughter is very much enjoying it.

I have only poked at my Sensor Project a little bit, but I was able to fix a bug that kept freezing my program after a few weeks. One of the sensors required an open and close when accessing it (it’s own custom version of it). But I didn’t close it when done with reading, so it kept pilling up open i2c connections until I couldn’t open any files at all in the program. I’m still testing to make sure it did resolve it, as it takes a few weeks for it to freeze up, but I’m fairly sure that was the issue, as it didn’t occur with any of my other sensors that were missing this one particular sensor.

Now that the installer is made for Darkstar, I’m thinking of adding additional support for sensors in my Sensor Project, but since I don’t physically have the new sensors, I won’t be able to test them, but I can at least get the backend set up and ready for testing. I’ll probably put these sensors under a “Un-Tested” section as a warning they may not function correctly.

That’s it for now. Slow progress for now due to the winter blues. Here’s hoping the sun decides to grace me with its un-clouded presence soon!

New standard Debian Installer!

I have been wanting to create a more standard installer for a while now, and I have finally gotten to it!
I decided to go with a Debian installer, as it was the simplest to figure out (after hunting down the right documentation), not to mention it’s one of the oldest ones. Originally I wanted to use a container installer like snap, AppImage, flatpack or Docker but .deb just seemed to make more sense for now…

So now Installing and Uninstalling has never been easier! Download and double click the file to install! This should make it easier for people used to Windows or Mac computers (but not compatible with Windows or Mac…). Once installed it should list kootnet-sensors in your Debian software manager. You can also remove with apt-get remove kootnet-sensors.

I have managed to get the old installer re-written in a way that will allow upgrades from the pre-deb installer to the new deb installer. It requires you to upgrade twice though, once to get the transition installer and another time to initiate the transition installer. Basically just upgrade from the web interface, wait 30 seconds, then upgrade again.

I have already done a fair bit of testing and optimizing, such as figuring out which apt-get dependencies are required and which are not. I put required to run at all stuff as required in the deb installer and sensor hardware requirements under recommends. This way you can use the management pages on Ubuntu for example, without requiring the hardware dependencies that may not be available on other Operating Systems.

After a bit more bug checking, optimizations, testing and documentation, I will release the new version and update the online documentation to match.

Sensor Control Enhanced!

I have finally finished up the Sensor Control section of the Web Portal. All configurations can now be edited and pushed to multiple sensors using Sensor Control. Once I do some testing, I’ll release this into the stable channel. Besides finishing up on the Configuration section of Sensor Control, I also added drop-down buttons for upgrading, power commands and misc other.

I would like to add the ability to review and add notes next, but I might space it out with a bit of reading. I got a new book “Modern Python Standard Library Cookbook”. I want to be able to do as much coding using the Standard Python Library as possible, so I can limit the outside requirements. This should also make my code cleaner and more efficient.

A new release is coming soon. As always, feel free to check out what is coming by upgrading to the development channel, but be warned, the development channel can be unstable.

Sensor Latency Tests Added

I finally got around to adding Sensor Latency testing. A button has been added under the “System Management” page to run it. The latency tests tell you just how long it really takes for a sensor to return a reading. This is required knowledge to set trigger variances in the seconds or less, so it doesn’t end up building a queue of reading requests waiting to get data. I have already set each sensor’s “multiple access wait times” to match more closely with how long it takes to get a sensor reading. This should improve performance when sensors are accessed at the same time.

Most sensors return data within 100ms (0.1 seconds). Some like the EnviroPhat can get a reading from each sensor within 10ms. But there’s one sensor, the light spectrum AS7262 that can take anywhere from 700ms to 8 seconds! I’m not sure exactly why, because my other spectrum sensors, like the BH1745 and the EnviroPhat, can grab readings in less than 20ms every time. I also checked the source code for the driver and how I set-up the API, and it’s set for under 30ms to get a reading for each of the six colours, so added up, it shouldn’t be much more than 200ms, and that’s assuming it can’t do them in parallel.

I have been putting off doing the configuration section under Sensor Control because I know I can actually get it to re-use my existing individual configuration pages without copying them … but it would take some creative re-working, and I have been lacking in creative motivations. Soooo… I think I’ll just copy-paste to get it working and refactor it later.

Kootnet Sensors Runs on Ubuntu

W00t, I made it so Kootnet Sensors can be run on Ubuntu (and probably other GNU/Linux systems). It just disables the hardware access functions but allows the web interface with access to “Sensor Control” to manage other sensors. Most changes were made to where the configurations are saved.

So this is awesome for testing! I can now just get it running from my Development Environment and test most HTML functions. It also does not require root access if there are no sensors.

Besides that, I also moved the sensor in use checks to the driver interface so nothing could bypass the check. The “Online Services” section of Sensor Control now works as well, except for setting the interval.

Things are coming along nicely.

Kootnet Sensors – Hardware Drivers Included

I have put the hardware drivers directly into Kootnet Sensors, instead of installing them with pip. This should ensure all the sensors work, regardless of the upstream project changes or updates I make to utilize new driver features. It took a bit of poking for the EnviroPlus sensor, but it and all other sensors I could test appear to be working well across the Pi Zero W, Pi 3b+ and the Pi4. I’m still wondering about including some of the other dependencies …

A few “bugs” have been fixed, such as making sure only one call to access a sensor is done at a time. Mostly around sensors that require a “keep-alive” thread, which bypassed the other in use checks. This should clear up errors around gas and the EnviroPlus’s particulate matter sensor.
I also made it so sensors that update multiple readings, even with single reading requests use a cache, so if the program is getting readings for all sensors (but through individual sensor reading requests), it can take from the cache for all requests made after the initial update within 1/2 a second.

The next thing I should really finish up is the Sensor Control section of the web portal. Once Configuration and Online Service sections have been done, it can pretty much do everything the PC Control Center app can do.

That’s it for now. Slow going for the short winter days and the long winter nights.

Kootnet Sensors Update

Now that Kootnet Network Testers are working well, I thought it was time to do a bit more on my Sensor Project.

I have started to integrate a version of each hardware sensor driver (the pip modules) into my program directly. The major reason for this, is I have already had 2 sensors change their functions in a way I had to fix my program to use it. This creates real problems because previous sensors don’t auto-update their drivers and conflicts start to arrive where it’s working on one but not another. I’m hoping I won’t have issues between the Pi Zero and the Pi3/4 due to the arm hardware differences. I’ll be sure to test it on all models before release, but so far the only one I am having any real issues with is the Enviro+. All the others have been entirely drop-in replacements. I think I have 4 sensors left to add drivers for, plus the Enviro+ I have to fix.
I’m happy to say that all the drivers so far have added up to about 500 Kbytes. So my installer is currently sitting at 1.5 Mbytes.

This change will also make the install much faster, as pip is not exactly a speed demon for downloading and install modules. I’m also considering adding other dependencies to the installer to make sure it will work, regardless of what happens with the dependencies own projects and even pip itself. But I’m also concerned about security with modules like Flask and the gevent since they are the internet-facing modules that you want to be patched and up to date. For now, I’ll most likely keep Flask and gevent as a pip install at the time of install, however that may change in the future… in fact, I suppose I could include the modules but then add a script to download the new version and copy it over the old one and restart the program… yeah that would probably work. I could even add an auto-update option for those modules at set intervals but disable it by default in case of issues introduced… actually… I can even have it update the module at the time of install.

Anywho, that’s what I’m working on now and once done, should improve consistent stability.

Network Testers Update

I was thinking about documentation when I realized I could integrate the instructions for using it, into the display itself. So to that end, I re-worked the hardware button operations.
With the initial release, each of the 4 hardware buttons was tied to a single command, with the exception of the 3rd button, which if pressed a second time, would initiate an update.
Now I have set the 4th button as a “Change Functions” button, that upon pressing will change the operations of the other 3 buttons, and at the same time, display a message on the screen for what each button does at the current function “level”. The change button currently has 3 levels it cycles through. The new button functions are as follows.
Primary Functions

  1. Run MTR tests
  2. Run iPerf 3 tests
  3. Nothing
  4. Change Button Functions

Secondary Functions

  1. System Information
  2. Standard Upgrade
  3. Development Upgrade
  4. Change Button Functions

Tertiary Functions

  1. Shutdown Remote Test Server
  2. Shutdown Local Unit
  3. Nothing
  4. Change Button Functions

With this change, I can tie more functions to the compatible Raspberry Pi hardware buttons. This should make it a much more useful mobile tool, especially when I start to add other types of tests it can run.

There are a few extra features, such as each instructional page viewed on the WaveShare display, now shows if the remote tester is Online or Offline. This same Online/Offline status is shown on the web portal as well.

The System Information page now shows the following for the ethernet and wifi adapter; Active/Inactive Status, DHCP or Static address setup and the current IP Address.

The about page now also displays the Primary Internet IP address along with the amount of free disk space in GB’s.

I also found and fixed a bug in the WaveShare display. If a button was pressed multiple times before the display finished its refresh, it would crash the display server. I fixed it by adding an “In Use” variable, and if “In Use”, any new commands will be dropped until no longer “In Use”.

I have already put the new update on the standard release channel.