Version 21.18 Released!

New Version out!  I was going to get pipenv working before release… but it looks like that’s going to take a bit more time.  
Here are a few highlights of this release.

  • Allow Live Graph on 3 point data
  • Merged Sensor SQL Interval & Trigger code & databases
  • Created Log viewer and downloader in Control Center
  • Tweaked System Report and added Installed Sensors to Config Report
  • Raspbian desktop and menu shortcuts added
  • Created uninstaller for sensors
  • Changed locations for sensor programs, configuration and data to be more in line with Gnu/Linux file structuring
  • Updated help files
  • Bug fixes and code optimization

Due to the location and database structure changes, this version is not compatible with older versions.  I recommend doing a clean install if you have  a previous version installed. 

Stuff

I spent some more time refactoring the Raspberry Pi Sensor code.  For one, I have the import code for each sensor, in the init part of the sensor access class, that’s created per sensor.  AKA, creating a class instance of a sensor, will then pull in the needed libraries at that time.  I did this originally so unneeded sensor libraries would never be imported, saving time and resources.  However, I then realized that I create the sensor object every time any sensor data is taken, which re-initializes the sensor every time!  Long story short, I moved the sensor access object instance creations out of the sensor retrieval functions and into the primary python program before the main loop.  It only initializes sensor access for sensors specified in the “Installed Sensors” configuration file.  This means sensor readings taken, will re-use the same sensor access instance that was created at the start of the program and only create access instances for installed sensors. 

Doc strings have been added to all functions and classes in the raspberry pi sensor code, and I started to get pipenv working for the package management and virtual environments for my programs… but it keeps erroring out on pipenv install.  It almost seems like its timing out on installing the packages, but I can’t be sure since I’m not sure where it stores the logs if it creates any.  In theory, it sounds like it will simplify the install process and help keep a consistent base to run on, but it looks like it will take a bit more time to have it function as intended.  

I was hoping to get pipenv working before creating new Gnu/Linux install scripts, and I was hoping to have the install scripts working before my next release.  So I’ll continue to diagnose the pipenv issues, in hopes that I can use it for with the new installers.  

Hope everyone’s weekend went well!

More SQL write modifications

I figured it was about time to re-look at the “write to SQL DB” code, especially for the datetime entries when writing to the database.  Long story short, I had SQL input the datetime values at the time of recording, but because the code grabbed a few readings to check against the variances, the data ended up being entering with fairly inaccurate datetime correlations (by hundreds of milliseconds or more).  

So, I changed the code in a few ways.  First, I found out how to get the exact same formatting for the datetime as SQL was using, and used that at the time of actually getting the sensor data (through Python datetime).  I also made the Trigger variance checks & database recording, work on 5 readings at a time.  If any of the 5 readings trigger a recording, all 5 will be recorded, so in theory, it will record entries before and after the event that triggered it. 

That’s it for today. 

Live Graph updated

w00t, the Live Graph now works with all supported sensors, AKA Live Graphs work on sensors that give 3 data points as well as just one. I also changed the layout to make it look “cleaner”.  

I have been thinking of a few other features to add, like LED / LCD screen support for displaying sensor readings on the sensors themselves, making the joystick on the SenseHAT functional and adding default reading offsets to the sensors themselves, and sending it with the sensor data for whatever reading it might be.  This will allow the Control Center to choose between the hardware default and a custom setting.

In other news, I have put a few more sensors around my house, to see how the temperature fluctuates in different parts of the house, during different times of the day and night.  The recordings should show me how the heat moves to different parts of the house, at what temperature and how long it takes to get up to temperature, and if certain rooms get more or less heat during the process.  With winter coming and living in a Mobile Home, there should be some good differences!

I have been testing some power systems for the Raspberry Pi, so that I can ultimately have a solar powered system, that charges a battery to keep it running at night.  I tried the “MP2636 Power Booster”, but had mixed results… it seems OK on my one Raspberry Pi 3B+, but it failed to stay on or even charge the battery for a few units…. I suspect polarity is wrong on my cheaper china batteries, but that doesn’t explain the other RP Zero W rebooting when swapping the primary power source.  Long story short, I don’t think the “MP2636” is going to work for my needs.  Next up is one made specifically for the Pi Zero, called “Juicebox Zero”.  It looks very promising, but I have to solder it on, so I’m putting it off for a bit.

** Few days later ** 

Guess I forgot to actually post this a few days back.  I have made some other major changes since the top half of this post.  For starters, I just did an overhaul of the write to SQL Database code.  I merged my 2 programs into one, so now both Trigger and Interval code run from the same python program.  I was mildly torn about doing this, but I think it’s ultimately for the best, not only on system resources and response times but for dealing with the readings as well, as you can now graph both Interval and Trigger readings from the database to a single graph page.  It’s also much nicer to deal with a single database for downloading and storing.

I have changed the “Clean” update code to run as a service (Manual start one-time run), this allows me to fully delete the main folders and files, without terminating the very program that’s doing it, which was preventing the re-install process after deleting everything… AKA it would delete everything and stop.  Now, the program initiates the update service to start, which keeps running after the program that started it, is terminated.  

What else was done… O yes, I added what sensors are installed in the Configuration report, worked on the Unit Testing and did a fair amount of refactoring.  I’m sure there were other things too, but I can’t think of them right now.  

Since I did a few overhauls in the code, I’m going to test things a bit more before an updated release.  

Remote Log Viewer & more Threading!

I got a remote sensor log viewer up and running, so you can live view the tail end of a log for individual sensors, or download all logs off all selected sensors and review them later.

I have also added threading to all HTTP downloads (logs + Databases) and Reports.  Reports generate a lot faster now!  I also made a minor change to HTTP downloading, which places the end IP oct at the start of the file for easier sensor identification on files.

Many parts of the programs are looking better and better, but there are some, like the network commands code, that just looks… lacking.  I have off and on been thinking about how to re-work my 30 or so ‘if + elif’ statements to check what command to run.  There must be a way to objectify the code, so it doesn’t look like a run on sentence… nothings coming to mind though, but then again, I haven’t eaten this morning yet… so maybe I’ll do that and re-think it over. 

I should really start working on the help file for the Control Center, since its starting to stabilize and there are enough core features I can document.  I would like to find a decent template file to use for my help files, since my versions are… also lacking.  That being said, they do get needed information across, in a non overwhelming way, they could just use a bit more “zing”.  

On the learning side of things, I’m pretty happy with progress.  I have learned a LOT of programing concepts and python libraries over a decently short amount of time.  As I go, and re-use some of those library and programing concepts, things just seem more “obvious” in how it all works together. When I first start on something new, I often pause on how to work it into my existing code, but once it’s there, it seems soo much easier re integrate into other sections.  After the 3rd or so time, I then start seeing how to refactor the code to be shorter, more obvious in function or more effective.  

Alpha.20.2 Released!

This release has a lot more nice touches, than huge features.

The reports section has been updated to include the following.
– Local Computer time when the report was generated
– Currently installed version on each sensor
– Last updated per sensor, with how it was updated at the time

When the “Advanced” features are enabled on the Sensor commands, it now turn the upgrade options into “Clean” upgrade options, which essentially deletes everything but data and config, and re-installs. 

The configuration window now has the live graph refresh time and a label above the DateTime entries to show how to set it (YYYY-MM-DD HH:MM:SS). 

I have also done a lot of changes on the RP Sensor install script, such as including Control Center and installing its dependencies.  I have made the HTTP upgrade method download a zip of the files and extract it, instead of getting each individual program file.  I also tested the install script until it worked flawlessly on a RP3B+ with a SenseHAT, a RP Zero W with a Garden HAT & a RP Zero W with a EnviroPHAT.  They all worked well, but I would NOT recommend using the Control Center on the Pi Zero, due to its limited resources (AKA its slow…).  

I have also updated some website links, pictures and downloads.  So it should be easy to find more useful information about the programs, especially with the new “Pictures & Screenshots” section.  

That’s it for now. 

Getting Raspbian working – KootNet Control Center

So I have been spending most of the day, getting the Control Center program to work on Raspbian, and installed with the normal Sensor installer script (AKA, you can use the Pi as a sensor AND controller by default).  

The dependencies are annoying, and matplotlib kept on failing to plot live graphs… 4 hours later I found out it’s some glitch in the version of numpy that was installed.  Long story short, getting the latest version fixed it.
python3 -m pip install -U numpy
It takes a pretty long time to update it this way on the Raspberry Pi Zero W, but I would rather that, then making the install process more complicated for the user.  

I’m doing final testing now, to make sure the single command line to install is working with no additional steps.  Once I get this working, I’ll probably release another version.

*Few hours later*
So the Raspbian installer script appears to work fairly well!  It installs both the sensor programs and the control center, then all dependencies and rebuilds the latest numpy.  It also puts a shortcut to the control center on the desktop of the pi user, for easy opening.  I set the first IP as the localhost 127.0.0.1 by default and checked it.  This makes it so you can start testing things right away on the local unit.  I tested the install script on a Raspberry Pi 3B+ with a Sense HAT and a Raspberry Pi Zero W with a EnviroPHAT.  The 3B+ went MUCH faster, and functioned pretty well … things worked just fine on the Pi Zero… but slow… just opening the Control Center takes about 30 seconds.  Still, it does function decently once its loaded, so it will make a good fall back way to test things. 

To give a bit of a timeframe to the slowness, I’m pretty sure it took about 2 hours on the pi zero one time, just to finish the install, most spent rebuilding the python3 numpy module.  I’m hoping to fix this easier in the future, by creating a virtual pip environment with specific versions.

Since I changed the install script to download a zip of all the files, then unzip it locally, its WAY FASTER!  It also seems less prone to network glitches.  Besides being faster, it also helps with sending less data, since the zip file uses compression.  

Lots more to enhance, but things are coming along nicely.  

I hope everyone has a great weekend!

Making programs to test my programs!

Another milestone reached!  I have now looked into and created a unittest program for testing the control center program.  It’s just a basic one right now, and it mostly tests the configuration sections.  As time goes on, I’ll add more tests and probably re-structure some of my other functions to be more test friendly.  

With tests coming along and the GUI broken up using OOP concepts, I think that covers 2 of the bigger things I had to figure out.  Now it’s just optimizing and expanding them… or another overhaul, if I find other new programing concepts worth implementing. 

I might work on adding the control center into the Raspberry Pi Sensor installer next … and creating menu shortcuts would be nice too.  That way, you can fully test out the sensor programs, simply by setting up a Raspberry Pi.  The sensors also responds a lot nicer, when it’s using 127.0.0.1 ^_^

So most likely I’ll be doing touch up work next.  I have a fairly decent list of smaller changes to be made, like making live graphs work with the Trigger sensor types, adding a “Last updated” entry on my reports and implementing more unit tests. 

Until next time.

GUI Refactoring!

I finally did it!  I have split up the GUI code into 9 files, one file per GUI Window for the most part. The GUI was in a single file over 1800 lines long, after breaking it up, most files are under 200 lines, but the Graph and IP Selection code is about 500 each.  Still, a lot nicer to deal with, then the 1800 line monster.  

It took me about 3 days to get that done, and there are still some little things to do, like clean up the shared configurations, since I had to mess around, trying to share config between the modular windows.  It appears to all be working now, but it will take a few more days to clear out old code and update other parts of the program to function more seamlessly with the new code.  I’ll also review the 500 line graph and IP selection modules to see about optimizing that.  

While I was breaking up the GUI, I also streamlined a few other things, like the logger, which has its own module now, so other windows can import and use that. There is MUCH less code in each module file for logging now.

Since there are no updates to the function of the program, I won’t be releasing a new version, but I did silently release ver. Alpha.19.2 last week, which includes a few new features I did before the GUI overhaul, like the Live Graph, Live Sensor Readings Report and multi threading for Sensor checks and downloads.

I want to make an overview video of the Control Center, but that will have to wait, since I don’t know much about Video production, and my program keeps changing 🙂  

Guess that’s it for today.  

More features!

It’s been a bit since my last update.  I’m happy to say, there are a few nice new features to the PC Control Center.  

I have now added the ability to graph a sensor’s reading in real time with a customizable duration between new readings.  I was originally going to pull up a graph for each sensor IP & selected sensor types, but matplotlib isn’t thread safe, and I’ll have to look into multiprocessing to get that working.  You can open multiple copies of the program itself, to view multiple live sensors at the same time.  

I also made a new Report, for viewing a snapshot of all sensor readings from the selected sensor IP’s.  Its a good way to see if you have the sensor setup right remotely, or to quickly test multiple sensors, making sure things are working as expected.  

I’m feeling pretty good about the advancements, but the primary GUI code is getting pretty big, at over 2000 lines of code!  The more that builds up, the less I feel like figuring out how to break it up, but it will be done at some point, and the longer I wait, the more I’ll of learned about coding, which should help with re-designing.  The main problem I’m having, is wrapping my head around how to interact with all the GUI components between software modules.  I’m sure the solution is in Object Oriented Programing, but I’m still working on OOP concepts, so I’m not sure what my timeframe is going to be for re-designing the GUI. 

O cool side note, I have 4 Raspberry Pi’s that have been up and running continuously for more than 60 days!  That’s pretty good for reliability, considering 2 of them are outside and I test my programs on them all the time. 

I almost forgot, I spent time making sure the PC Control Center works properly under Linux, specifically Raspbian.  I had to add some customizations in the program for the GNU/linux OS, especially in the GUI window sizes, but it works very well now!  I’m going to put the PC Control Center into the Raspberry Pi Sensor installer, so you can manage multiple sensors, from the sensors themselves through VNC.  This may help with not so great remote connections, as the local units should have a better connection to interact with the local sensors and prevent network timeouts.  

On another side note, I’m kinda excited thinking of what I might do with the sensors, using these programs as a interface to get readings remotely.  I was thinking of making other programs, that interact with the sensor network server, to get readings and combine a bunch to automate certain things, like opening doors or windows, turning on and off lights, adjusting light levels based on current light levels, etc.   
With halloween coming up, one could add a thermal or motion detector with a distance sensor to perfectly time spooky whatevers. Imagine a skeleton that does something different depending on the distance of the person or objects, for example at 6 or more meters, it plays something to get them over, like a nice lady voice offering candy, then at 3 meters, it starts to laugh, if the person moves away, it stops laughing, and tries to call them back, otherwise if they keep coming, it laughs until 1 meter, then it does some scary thing with lights and all.
Loads of cool stuff that can be done!  

I’m sure I’m forgetting to mention a bunch of other changes …. like how I multithreaded the database downloads, but that’s good enough for this update.  I plan on doing a bit more testing and refactoring before putting out another version.  

Until next time!