Alpha.22.24 Released

New version out!  According to my website, it’s been exactly 1 month since the last release.  There have been a lot of changes since the last one, which includes a change from python sockets to Flask.  So this version is very much not compatible with the previous, as far as the Control Center communication goes.  The bigger changes include the following.

  • Change from Sockets to Flask
    • Starting WebApp to control sensors with a web browser
  • Enabled support for 2 new Pimoroni sensors
  • Sensor Configuration now allows Config & Installed sensors to be adjusted
  • Added 6 channel support to the colour spectrum recording and Live graphing
  • Added sensor provided temperature offset, determined by installed hardware.  This is also recorded to the database
  • Refactoring, bug fixes & tweaks

One thing that is missing is the ability to do a Plotly graph with all 6 colour channels, but it still graphs the RGB of a 6 channel sensor just fine. I have also changed the python virtual env, included in the windows installers, so you can’t do an upgrade from previous versions.  The RP sensors should upgrade just fine though. 

That’s it for now.  

Updates!

I added support for my new sensors, and got the live graph to show all 6 colours of the new 6 channel spectral sensor.  

All good news, but I feel like I’m starting to get a bit lazy on my implementations, and I need to set aside some time to refactor.  I have already started a bit here and there.  For example, I have been moving stray configuration options and settings into the config python file.  I want to make it the one-stop file for almost all the customized variables.  I also need to get my unit tests working again, since the move to Flask.  

There has been a lot of additions since the last release now, so I’m trying to work on tests instead of features … but I do kinda want to get the SQL note viewer and editor working.  I’m pretty sure no one actually uses my program yet, so I might hold off on the release and work on a few more features and some refinement.  

In other detailed adjustments, I was thinking of how to add the different Wavelengths of light into the database.  Since I have RGB saving into the database as individual colours “Red, Green, Blue”, I finally decided on adding the 6 common colours of the visible spectrum and make notes in the sensor module for the Wavelengths it’s capturing.  This should allow a balance of accuracy and simplicity.  I was thinking of using the Wavelength itself as the entry column but realized it would add extra complications down the road.  

I have definitely slowed down on progress, and I think the winter blues are causing a down turn of motivation.  

New Sensors and work

So, good news first, I just got some new sensors as of … 30 min ago?  I’m really excited to test out the 6 channel spectrometer, but I also got a light and proximity sensor, Garden HAT for the Pi Zero and a few LCD displays.
Specifically, I got the following from Pimoroni.

  • AS7262 – 6 channel spectral sensor
  • LTR-559 – Close range (4 cm?) proximity detector & lumen sensor
  • 1.12″ Mono OLED 128×128 display

In other news, I revisited my Raspberry Pi Network Testers, which are 2x Pi’s made to test Ethernet cables and routes.  It’s now been updated to my present Python & Linux programming knowledge.  AKA it now uses systemd services, bash installer scripts and works directly with the original E-Ink libraries, instead of a 3rd party one I found.  All in all, its cleaner, more efficient and functions as one would expect for use and errors.  

I’m getting closer to releasing another Sensor Control Center update, but I might tweak a few things before that.  I want to enhance my SQL note section, so you can browse, edit and delete entries.  
We’ll see what happens. 

I have also been thinking about making an I.T. technician toolkit of sorts.  Creating a simple GUI app to quickly do common IT related tasks, like quick virus scan with a self-contained scanner, ccleaner type clean-ups, show system information like IP, uptime, free mem and HDD, etc.  I think I’ll also create a built-in script runner, so you can pick from a dropdown to run whatever script you like, as well as the ability to add new ones.  If I create most things as scripts, I should be able to do almost anything … So I’m kinda looking forward to releasing a useful diagnostic tool.  

Since I have been a bit busy with work, I have not been working as much on my programs.  

Until next time!

Minor changes & stuff

I have been working to update the smaller stuff after those overhauls before.  Editing comments, updating help files and fixing bugs.  

Now that I have Flask in use, it allows me to show reports off the sensor itself, by going to a URL with the sensors IP (or hostname if DNS is setup right).  I’m working on http://sensor_ip:port/TestSensor.  I created a combo of all 3 Control Center reports in one.  Going to that URL with an actual IP and the right port would give a nice combo HTML report of the sensor, complete with name, version, current sensor readings, configuration setups, etc!  
The big thing I like about this is you can take your cell phone, make it a wifi hotspot the sensors connect to and goto the URL in a web browser on said phone to see its status.  If I make an actual phone app, I may be able to automate the process with a push of a button.  Cool possibilities for the future! 

I have been pondering some other back-end changes.  Since I know a lot more about python now, I might move a bunch of my install code from the bash script into the program’s first launch.  I should probably keep the bash script to a bare min, just enough to get the program needs installed, then let the program start and take over from there.  

I should probably set a time to do a purely refactoring session too.  There are a lot of places where the code could use improvement where I was trying to get it working at all.  I also really want to get the Live Graph working with threading for the data gathering.  Since right now, if the connection is not so great, the program will freeze until it times out.  So what I’m thinking of doing, is simply threading the data gathering and putting results in a queue or pool of sorts.  So long as this pool allows the program to continue while waiting for data, it should work very well.  Otherwise, I’ll set aside some time to really think about how to make it shine … or maybe replace it… Since Matplotlib is not thread safe to begin with, maybe I should look for something that is more like a multi-camera dashboard.  I’m pretty sure most people are going to want to put a dashboard of live sensors up in front of them, to monitor things like crops and such.  

I will most likely release a new version soon, as there are a lot of nice touches on this one, such as most areas are threaded to prevent program freezes now (minus the live graph), the program is no longer dependant on a 3rd party HTTP server, the Control Center sensor re-configuration section has been re-done to allow config and installed sensor changes. It also allows grabbing a sensor configuration to compare or re-distribute to other sensors.  Graphing now allows a default temperature offset from the sensor itself, which has an automatic offset based on the hardware chosen, as well as allows a custom entry.  Notes can now be entered into the Sensor Database with a date and time stamp so it will (eventually) show up in graphs beside relevant sensor data. I have created an update and program checks that will make needed adjustments to settings and files after an upgrade.  Hmm, what else… O yeah, I was able to remove the rebuild of numpy with the new Raspbian version that’s out, but I had to add ‘atlas’ i think… but just for install, not rebuild it, so that’s good.  

Lots of good improvements and it seems to be running fairly smoothly with the transition from sockets to Flask… although I must admit, I do notice a slight more delay using Flask, which makes sense due to the overhead.  The delay is also measured in ms not seconds, so its not bad. It is especially hard to notice now that I have added threading to pretty much every sensor reading grab, so it shows up as a delay in showing, not a frozen program.   

That’s it for now.  

Flask seems a good fit

I was looking for a better way to do my Network Server commands, so I headed over to the Python IRC channel and asked for some better ways to send text across the network.  After some chatting back and forth someone mentioned HTTP and REST, so I started my research.  It didn’t take long to find Django (what I used before) and Flask (I saw before, but haven’t used).  So I looked around trying to see what the purpose and differences of the 2 are.  It was obvious pretty quickly that Flask was the one.  My commands and data are pretty basic, so having a micro framework to send and receive made sense.  There also seems to be a lot of module attachments to enhance Flask, so future expansion is there.  

I have now replaced my basic sockets server with Flask.  I have to admit, there’s a lot to like.  For one, I can just go to the URL of the sensor in a web browser to view results!  Soo very helpful during testing to see if anything looks off.  This also opens up possible phone apps that just use a wrapper around a web browser.  It would be pretty sweet to use my phone to interact with sensors.  

Definitely a few more possibilities in the near future.  I still have to rework some of my unit tests and actually test the program a bit more, since I just overhauled the network service, but so far it’s looking pretty good.  

Lots to do, so back to it I go. 

Update system & notes!

I have been working on a few things as of late.  For one, notes can be saved into the sensor Database at whatever time you choose.  This is meant to aid in putting notes beside the actual sensor data.  I’ll be adding the notes into the graphs later, so that they will show up with the graphed sensor data, using the time you enter!  AKA, you notice the sensor malfunctioning at a certain time, you enter a note with the same or close Date & Time saying “Malfunction, ignore this” or “Very important data represents time travel”.  Whatever helps identify what the data represents.  It can also be used to bookmark or tag certain times with things, like manual settings for temperature offset or other important information not being captured by the sensors, etc.  Later on, I’ll try to get it so you can “Graph by notes”, and it will graph say, 1 hour on either side of every note’s time frame, making it easy to instantly populate graphs with important data without having to search every time. I’ll be giving it more thought for sure. 

Since I seem to like ordering my options in the config and sensor files, in a way that’s not great for compatibility (AKA I don’t add every new option to the bottom, I reorder the whole list).  This is a pain, as I manually have to log in through SSH and re-adjust the sensor configuration files for the new options.  So I had a few ideas on solving it. 
1) Add the ability to change installed sensors from the control center
and or
2) Create some updated functionality to automatically make needed changes after the upgrade. 
I went with the 2nd option first and made an upgrade module that adjusts any needed files and related settings, then re-saves them.  It also uses the versioning system, so it can incrementally update from this version onwards, even if you miss a few versions along the way!  This will make my future upgrade tests go a bit smoother!

My other major upgrade was to the environmental temperature adjustments.  Before it was a manually set option in the graph window, but this makes it annoying to swap between different sensors with different offset needs.  So I moved the offset into the sensor itself and created different offsets for different setup combinations.  This way it’s all automatically set by how you set your installed sensors configuration file!  AKA it will give recommended offsets depending on what hardware is being used.  So now you can check a “Use Default” temperature offset checkbox, and the graphs will use the offset provided by the sensor.  A custom offset is also available in the sensor configuration file, in case a sensor doesn’t follow standards.  And of course you can uncheck the use default and set your own setting during graphs.  Whatever is set on the sensor, will also be saved into the SQL database when the environmental temperature readings are saved.  

Those are the 3 main things I have been working on.  

I think I’ll re-work my re-configuration of the sensors.  I think I can make a decent live update by pulling and pushing the text files into and out of a textbox, similar to my Log viewer and Notes window.  If I do that, I think the program can do everything remotely!  Between the log viewer, update module and soon to be installed sensors and config adjuster, keeping my demo and test sensors running smoothly through upgrades is starting to sound like an easier task 🙂

Nicer touches

I have re-visited my Sensor Readings Report code and changed it to create all the readings in a table for lining up.  I also put each sensor into its own bordered box to help separate sensors.  While I was doing that, I figured out how to get multiple packets of data with sockets.  Before, all my data was under 4096 bytes, except for my logs, which I only needed the end of anyway.  However, I can now get a bit more of the logs and ensure data is properly received. I had some trunicated errors doing Reading reports randomly before and it caused the socket to die, requiring a timeout.  Now it functions as intended and no delay in restarting occurs.  I also added the ability for the socket to be rebonded right away if the application is terminated unexpectedly.  This should speed up Sensor Command restarts after remote upgrades.   

I have now set up a Pi + Goal Zero Sherpa 50 with a solar panel keeping it charged in the window.  This should give me a good idea if it will work as a self-sustaining power system for off the grid sensors.  I’m told windows can block a lot of sun rays, making solar charging less efficient, so if it works there, it will work even better outside.  I still have to figure out how to keep the snow off the solar, but I do have an idea… angled setups and mirrors would probably allow it.  Multiple mirrors set up in different spots could also help with “temporary intensive charging”, to help give it a boost at certain times, which could help during long dark periods… I’m just not sure how long they will be effective before I have to adjust the positions of the mirrors, due to sun path changes through the seasons.  

Feeling pretty good about the progress again.  It was nice to take a bit of a break, but I would like to try and do at least a bit of programing every 2nd day to maintain forward momentum.  

Hope everyone had a great weekend!

SenseHAT Ehancements + Updates

If you are lucky enough to have a SenseHAT with your Raspberry Pi, I have updated the sensor software to allow the use of its JoyStick and LED grid! The following commands are

  • Up = Scroll IP address on the LED grid
  • Down = Display “Steve” on the LED grid OR Confirms Shutdown
  • Left = Scroll humidity
  • Right = Scroll Temperature
  • Push = Initiate Shutdown (Requires confirmation of Down JoyStick)

Due to the extra additions on the SenseHAT, it is a nice sensor to start with for trying things out, since you can just plug in power and be able to see readings right on the LED grid AND have it shut down with the button combo.  Other sensors you have to shutdown remotely with the Control Center or login directly and shut down, and of course, no other currently supported sensor has a display. 

I now have a working Gnu/Linux installer working for both SMB and HTTP, but customizing a file is required if you want the desktop shortcuts on a different user than pi (Menu shortcuts work just fine). 

In other news, a few new battery backup systems have come in for me to try out.  This time I decided to go with something a bit more expensive to help with the solar deployments.  To that end, I grabbed a “Goal Zero Yeti 150” and a “Goal Zero Sherpa 50”.  They both have hookups for solar panels and are pretty hefty.  I put a RP Zero + Garden HAT on the bigger 150 to see how long it lasts.  I’m going to put the smaller one outside with a solar panel and see how that goes.  Both units oddly enough cost me about $250 each.  If they work as fully self-sustaining units, they are worth the price.  O yeah, the manual for the Yeti was pretty useful! It says to help with using the battery in colder temperatures, put it in a cooler, as the ambient temperature it puts off would keep it warm enough inside the insulated container.  A very good idea, since it will be in the outdoors during winter and I have been thinking of ways to protect the battery units from the elements.  I look forward to testing results, but I suspect I’m going to have trouble keeping the snow off the solar panels … I might have to be a bit more creative for that.  

That’s it for tonight.  

Version 21.18 Released!

New Version out!  I was going to get pipenv working before release… but it looks like that’s going to take a bit more time.  
Here are a few highlights of this release.

  • Allow Live Graph on 3 point data
  • Merged Sensor SQL Interval & Trigger code & databases
  • Created Log viewer and downloader in Control Center
  • Tweaked System Report and added Installed Sensors to Config Report
  • Raspbian desktop and menu shortcuts added
  • Created uninstaller for sensors
  • Changed locations for sensor programs, configuration and data to be more in line with Gnu/Linux file structuring
  • Updated help files
  • Bug fixes and code optimization

Due to the location and database structure changes, this version is not compatible with older versions.  I recommend doing a clean install if you have  a previous version installed. 

Stuff

I spent some more time refactoring the Raspberry Pi Sensor code.  For one, I have the import code for each sensor, in the init part of the sensor access class, that’s created per sensor.  AKA, creating a class instance of a sensor, will then pull in the needed libraries at that time.  I did this originally so unneeded sensor libraries would never be imported, saving time and resources.  However, I then realized that I create the sensor object every time any sensor data is taken, which re-initializes the sensor every time!  Long story short, I moved the sensor access object instance creations out of the sensor retrieval functions and into the primary python program before the main loop.  It only initializes sensor access for sensors specified in the “Installed Sensors” configuration file.  This means sensor readings taken, will re-use the same sensor access instance that was created at the start of the program and only create access instances for installed sensors. 

Doc strings have been added to all functions and classes in the raspberry pi sensor code, and I started to get pipenv working for the package management and virtual environments for my programs… but it keeps erroring out on pipenv install.  It almost seems like its timing out on installing the packages, but I can’t be sure since I’m not sure where it stores the logs if it creates any.  In theory, it sounds like it will simplify the install process and help keep a consistent base to run on, but it looks like it will take a bit more time to have it function as intended.  

I was hoping to get pipenv working before creating new Gnu/Linux install scripts, and I was hoping to have the install scripts working before my next release.  So I’ll continue to diagnose the pipenv issues, in hopes that I can use it for with the new installers.  

Hope everyone’s weekend went well!