Author Archives: chad.e

Kootnet Sensors Update 0.35.142

I have been testing Kootnet Sensors and fixing minor issues and making other minor changes. So far nothing major has come up so I’m pondering making this version 1.0 after a bit more testing.

Some of the changes include.

  • Clearing validated and failed login sessions after user/pass update
  • Removing overly noisy debug logging for cleaner logs and performance
  • Updated debug logging for clarity in some situations
  • Fixed continual DB writes for “Normal” values in High/Low trigger recording
  • Fixed “null byte” warning in bash script during install
  • Changed the update server to use HTTP instead of HTTPS due to a bug in the auto-update server downloading (I’m still not sure what is causing it, since manual updates still work using HTTPS)

I’m pretty happy with this version. There are some nice updates since the original release of 0.35.114 that include performance, bug fixes and security updates, not to mention the networking overhaul.

This version is in the Stable channel release now and will be in the master branch of GitHub shortly.

Kootnet Sensors Networking

It is complete! The networking configuration overhaul that is. Networking before this change only allowed you to configure a single wireless network to connect to and set it as either DHCP or Static for its IP address. It also used a ‘template’ of the wpa/dhcpcd system configuration files, so if you manually changed any settings in the OS config files, it would overwrite them. This also had the downfall of not adjusting for any future default settings that may be used in Raspbian.
Those downfalls have now been rectified! I have made Kootnet Sensors load the OS wpa/dhcpcd files and search for the specifics it’s going to do, adjust them and leave everything else intact! This means you can manually edit the files without worry, and also allow future default changes from Raspbian itself. Networking can now do the following.

  • Edit ALL network adapter IPv4 settings (minus the loopback adapter)
    • The network settings on each adapter will auto-populate if they are available, showing IP/Gateway/DNS/etc.
    • Changing between adapters will refresh network settings for each adapter
  • Add/Edit/Remove wireless networks to auto-connect to, allowing, for instance, auto network connectivity at the site you are recording data and when you bring it back to your shop.

There are some other minor changes I’ll be looking into, such as seeing if I can set priorities for the wifi connections, allowing for example the sensor unit to prioritize connecting to a laptop with an AD Hoc wifi network configured over the usual wifi, letting someone easily connect up for diagnostic purposes.

The new version is in the Development channel, so feel free to check it out, it should be pretty stable.

I’m not dead!

It’s been about half a year since my last post, but I’m not dead! I did run into some family and personal issues, so I have taken a lot of time to get some of that sorted out. I’m happy to say I should be getting back to programming soon. I have been doing a bit of programming during that 6 months as well but I kinda forget most of what was done. When I was programming last, I was re-working the networking in Kootnet Sensors so you can adjust multiple adapters for the IP address and also add/edit multiple wireless connections. This way you can have your sensors connect to different available networks. It’s about 1/3 done now as you can already edit all network adapter IP addresses.

I just wanted to do this quick post to let people know I’m still here and plan on doing more work on both Kootnet Sensors and the Raspberry Pi Network Testers.

Kootnet Sensors 0.35.114 Stable!

My tests went well and I have only been tweaking log messages and basic stuff like that, so … out with version 0.35.114! You’ll notice it’s 0 instead of Beta in the front, as I’m expecting to change it to 1.0.0 on the next release! … well most likely. Anywho, there’s a lot of behind the scene changes as well as some more obvious user changes.

The big changes since the previous release (that I can remember off the top of my head) include

  • Live graphing for local and remote sensors
  • Revamped upgrade system (HTTP/SMB support for Auto Upgrades, MD5 checking, version checks)
  • Enhanced security of user logins
    • Passwords are now salted and hashed using python’s built in hashing “scrypt” instead of MD5 (much more secure)
  • Major performance improvements to web interface (more threading & optimizations)
  • IP ban list for too many failed logins
  • Security updates as per GitHub’s CodeQL scanning
  • Bug fixes

Install or upgrade and give it a try! I’ll probably take a break for a bit now. Next, maybe I’ll put more serious work into making the physical Tri-Corder case.

Upgrade System Overhaul

While I was going through and testing things, I found my Kootnet Sensors upgrade system was in need of an overhaul. For starters, the SMB (Windows Shares) version of updating was more or less a hack job to use in my development environment. It had no saveable options and settings were simply hardcoded in. No more! It can now be configured and even used in automatic upgrades! Here’s a quick list of changes and updates

  • SMB Upgrade support (beyond my personal use)
    • Options that can be set
      • Login Username / Password
      • SMB Server + Share address
      • MD5 file verification support
      • Automatic upgrade support
  • When doing a “Clean Install” (AKA Re-Install from the program itself) it downloads full installers
    • Full installers contain required and known to work Python modules
  • Added the ability to disable upgrade file MD5 verification
    • I don’t recommend disabling MD5 verification, as it detects bad downloads which could render your install inoperable
    • This is more for my use when I’m lazy and don’t update MD5s during development
  • Added safe guards to make sure multiple upgrades don’t run at once, including others like OS or python module upgrades
  • Improved web interface upgrade messages
    • If something goes wrong during install of an update (like trying to install the same version), the notification of upgrade in proccess will go away once the upgrade script is no longer running
    • When automatic upgrades are disabled, web notifications of updates now links to the correct upgrade method (SMB/HTTP)
  • Check required update files on update server will now check SMB/HTTP depending on auto update selected type
  • Updated generic configuration save/load to allow ‘=’ sign use in options without breaking configuration load for said option
  • Added automatic python module install on program start
    • This is more of a ‘just in case thing’, but should you download the source and just run the program, in theory it will automatically install what you need and start running. HOWEVER, there are some linux OS packages that are required for some functionality and you still need to have Python3 + pip installed prior.
    • This is also not the recommended way to install from scratch, see GitHub readme for more information
  • Added warning message if default username/password are not changed
  • Fixed listed security issues in GitHub code scanning
  • Updated IPv6 support
  • BugFix: Ignore update version when doing a clean-install

The upgrades overhaul took longer than it should have because I wasn’t super happy with the structure of it and re-did it a few times. Next time I really have to write it down and go through the thought processes before writing code. Anywho, I’m pretty happy with it now after creating 2 python classes to interact with upgrades and new available versions.

I need to do some more thorough testing before a new release but I expect it should be out in a few days. This won’t be the 1.0.0 release but should things go smoothly, the next one might be!

Kootnet Sensors Graphing

I’m decently satisfied with Kootnet Sensor’s Live Graphs now. I added a few new features, for one, you can set how many data points are plotted before it starts to remove old readings. This gives it a “scrolling effect” once you hit the max data points. It also helps with performance and viewing clarity. I also added a “Performance Mode” to disable animations, and later on, other performance-enhancing tweaks.

Database Graphing (Plotly) is now being used for Graph emails and a configuration for Database Graphing has been created. Email Graphs uses its own copy of the configuration so you can have different settings for general graphing and email graphs. It even works if you select an MQTT sensor for graphing instead of the local sensor.

I’m kinda thinking of making the next release version 1.0.0. Additional planned changes are more enhancements to existing sections, so I think the base is pretty stable now. I’ll see how it goes and do some more testing and decide later on.

That’s it for now, check out the new Live graphing, it’s pretty sweet!

Kootnet Sensors Live Graphing!

Some exciting news, Kootnet Sensors now has Live Graphing! This has been something I wanted to add since I first moved to a web-based app, however, I had trouble with the JavaScript syntax using Charts.js. Now that I have done more work with JavaScript, HTML and CSS, I finally figured it out and got it working!

By default, live graphs are set to graph the local sensor but I have added options to allow it to graph remote sensors too! Options include

  • Select which sensors to graph
  • How many sensor graphs are displayed per row
  • Update Interval in seconds
  • Sensor address (leaving this blank will graph the local sensor)
  • Disable SSL Verification, which is needed when graphing remote sensors that have a self-signed SSL certificate or if you are using an IP address.

Another neat ability with Live Graphing is you can open up multiple windows to view different sensors (So long as Disable SSL Verification is not enabled).

I am still finishing up work on Live Graphing but expect a new developmental release by the end of the week.

More updates!

More changes to Kootnet Sensors as of Beta.35.20. Here’s a quick list.

  • BugFixes
    • Sense HAT joystick service runs properly now (was continually “crashing”)
    • Datetime conversions work properly now (based on UTC offset)
    • Updates to version numbers that add up to only 9 characters works correctly
  • Ban IP from logging in temporarily when 10 or more logins are unsuccessful within an hour
  • MQTT Subscriber & Checkins Server pages that contain a list of their sensors, is now generated by button and in the background
  • When Demo mode is enabled, logins are no longer required (but most system commands are disabled)
  • Updated install package to work with Debian Bullseye & Python 3.9
  • Replacing sensor databases now verifies the uploaded database is of the correct type
  • Other minor changes and updates

Kootnet Sensors

The update system is now good to go! All HTTP updates are downloaded through HTTPS and do MD5 checks to make sure the download is okay.

I have created a URLs configuration where you can adjust the server address for program updates & the sensor check-in server if you ever want to configure your own servers.

I have added a system-wide alias to start up Kootnet Sensors Terminal Configuration Tool from the command prompt, which is ‘ks-tct’ without the quotes. by default, it runs using the python virtual environment meant for upgrades, because of this, you should be able to use the TCT even when there’s something wrong with the main virtual environment or program (such as missing python modules, which you can re-install with ks-tct).

The Login system has been completely overhauled. I’m using sessions with flask now, and when someone logs in successfully, a login ID is given (randomly generated SHA256). Due to this change, Remote Management is no longer compatible with previous versions (at least not for parts that require authentication). So I have bumped the version to Beta.35.x. This is also good since the username/password has to be reset due to a change in how user/passwords are saved (changed previously).

Other things that have been changed include

  • Seperated sensor offsets to it’s own configuration (only environmental temperature, more to come later)
  • Threaded initial cached variables update (HTTPS server loads much quicker)
  • Instructions and file checks added for changing update server URL
  • Refactoring to remove the need of some 3rd party Python modules (Flask-HTTPAuth, dateutil)
  • Code Clean-up

That’s it for now. You can try it out by updating to the latest beta version.

General Updates

I have not decided on what parts to use for the Raspberry Pi tricorder but with the release of the Raspberry Pi W 2, it looks like it could be a fine choice!

In Kootnet Sensors news, I have been enhancing and refining sections. For instance, I’m using Python’s scrypt to create password hashes. It’s much more secure, especially due to adding a randomly generated “salt” to the user’s password that’s generated every time a password is changed. Unfortunately, I can’t convert old passwords to the new format, so it just re-sets the username/password to default. Normally I would never dream of doing that to a userbase, but I don’t really have a large user base per se… so reset!

I am now working on re-doing my whole update system. It’s actually pretty much done now, I just need to tweak it a bit and test some more. With the updates being done in Python instead of bash scripting, I was able to add some nice features, such as MD5 checksum verification of downloaded update files, which, should prevent bad downloads from breaking the install. This also allowed me to down 3 bash scripts to just 1 Python script and 6 Linux service files to just one. I also have the Python update script run from a separate Python virtual environment, which allows a “clean install” to be done (remove all program files and main Python virtual environment before re-installing).

There are a few other things I wanted to do as well, such as changing the Web Portal login from HTTP basic to something a bit more modern.

I am trying to make it so the next release will be version 1.0.0, as a lot of the program is pretty decent as far as the backend goes. Guess we’ll see if that happens 🙂

Until next time!