New Sensors and work

So, good news first, I just got some new sensors as of … 30 min ago?  I’m really excited to test out the 6 channel spectrometer, but I also got a light and proximity sensor, Garden HAT for the Pi Zero and a few LCD displays.
Specifically, I got the following from Pimoroni.

  • AS7262 – 6 channel spectral sensor
  • LTR-559 – Close range (4 cm?) proximity detector & lumen sensor
  • 1.12″ Mono OLED 128×128 display

In other news, I revisited my Raspberry Pi Network Testers, which are 2x Pi’s made to test Ethernet cables and routes.  It’s now been updated to my present Python & Linux programming knowledge.  AKA it now uses systemd services, bash installer scripts and works directly with the original E-Ink libraries, instead of a 3rd party one I found.  All in all, its cleaner, more efficient and functions as one would expect for use and errors.  

I’m getting closer to releasing another Sensor Control Center update, but I might tweak a few things before that.  I want to enhance my SQL note section, so you can browse, edit and delete entries.  
We’ll see what happens. 

I have also been thinking about making an I.T. technician toolkit of sorts.  Creating a simple GUI app to quickly do common IT related tasks, like quick virus scan with a self-contained scanner, ccleaner type clean-ups, show system information like IP, uptime, free mem and HDD, etc.  I think I’ll also create a built-in script runner, so you can pick from a dropdown to run whatever script you like, as well as the ability to add new ones.  If I create most things as scripts, I should be able to do almost anything … So I’m kinda looking forward to releasing a useful diagnostic tool.  

Since I have been a bit busy with work, I have not been working as much on my programs.  

Until next time!

Minor changes & stuff

I have been working to update the smaller stuff after those overhauls before.  Editing comments, updating help files and fixing bugs.  

Now that I have Flask in use, it allows me to show reports off the sensor itself, by going to a URL with the sensors IP (or hostname if DNS is setup right).  I’m working on http://sensor_ip:port/TestSensor.  I created a combo of all 3 Control Center reports in one.  Going to that URL with an actual IP and the right port would give a nice combo HTML report of the sensor, complete with name, version, current sensor readings, configuration setups, etc!  
The big thing I like about this is you can take your cell phone, make it a wifi hotspot the sensors connect to and goto the URL in a web browser on said phone to see its status.  If I make an actual phone app, I may be able to automate the process with a push of a button.  Cool possibilities for the future! 

I have been pondering some other back-end changes.  Since I know a lot more about python now, I might move a bunch of my install code from the bash script into the program’s first launch.  I should probably keep the bash script to a bare min, just enough to get the program needs installed, then let the program start and take over from there.  

I should probably set a time to do a purely refactoring session too.  There are a lot of places where the code could use improvement where I was trying to get it working at all.  I also really want to get the Live Graph working with threading for the data gathering.  Since right now, if the connection is not so great, the program will freeze until it times out.  So what I’m thinking of doing, is simply threading the data gathering and putting results in a queue or pool of sorts.  So long as this pool allows the program to continue while waiting for data, it should work very well.  Otherwise, I’ll set aside some time to really think about how to make it shine … or maybe replace it… Since Matplotlib is not thread safe to begin with, maybe I should look for something that is more like a multi-camera dashboard.  I’m pretty sure most people are going to want to put a dashboard of live sensors up in front of them, to monitor things like crops and such.  

I will most likely release a new version soon, as there are a lot of nice touches on this one, such as most areas are threaded to prevent program freezes now (minus the live graph), the program is no longer dependant on a 3rd party HTTP server, the Control Center sensor re-configuration section has been re-done to allow config and installed sensor changes. It also allows grabbing a sensor configuration to compare or re-distribute to other sensors.  Graphing now allows a default temperature offset from the sensor itself, which has an automatic offset based on the hardware chosen, as well as allows a custom entry.  Notes can now be entered into the Sensor Database with a date and time stamp so it will (eventually) show up in graphs beside relevant sensor data. I have created an update and program checks that will make needed adjustments to settings and files after an upgrade.  Hmm, what else… O yeah, I was able to remove the rebuild of numpy with the new Raspbian version that’s out, but I had to add ‘atlas’ i think… but just for install, not rebuild it, so that’s good.  

Lots of good improvements and it seems to be running fairly smoothly with the transition from sockets to Flask… although I must admit, I do notice a slight more delay using Flask, which makes sense due to the overhead.  The delay is also measured in ms not seconds, so its not bad. It is especially hard to notice now that I have added threading to pretty much every sensor reading grab, so it shows up as a delay in showing, not a frozen program.   

That’s it for now.  

Flask seems a good fit

I was looking for a better way to do my Network Server commands, so I headed over to the Python IRC channel and asked for some better ways to send text across the network.  After some chatting back and forth someone mentioned HTTP and REST, so I started my research.  It didn’t take long to find Django (what I used before) and Flask (I saw before, but haven’t used).  So I looked around trying to see what the purpose and differences of the 2 are.  It was obvious pretty quickly that Flask was the one.  My commands and data are pretty basic, so having a micro framework to send and receive made sense.  There also seems to be a lot of module attachments to enhance Flask, so future expansion is there.  

I have now replaced my basic sockets server with Flask.  I have to admit, there’s a lot to like.  For one, I can just go to the URL of the sensor in a web browser to view results!  Soo very helpful during testing to see if anything looks off.  This also opens up possible phone apps that just use a wrapper around a web browser.  It would be pretty sweet to use my phone to interact with sensors.  

Definitely a few more possibilities in the near future.  I still have to rework some of my unit tests and actually test the program a bit more, since I just overhauled the network service, but so far it’s looking pretty good.  

Lots to do, so back to it I go. 

Update system & notes!

I have been working on a few things as of late.  For one, notes can be saved into the sensor Database at whatever time you choose.  This is meant to aid in putting notes beside the actual sensor data.  I’ll be adding the notes into the graphs later, so that they will show up with the graphed sensor data, using the time you enter!  AKA, you notice the sensor malfunctioning at a certain time, you enter a note with the same or close Date & Time saying “Malfunction, ignore this” or “Very important data represents time travel”.  Whatever helps identify what the data represents.  It can also be used to bookmark or tag certain times with things, like manual settings for temperature offset or other important information not being captured by the sensors, etc.  Later on, I’ll try to get it so you can “Graph by notes”, and it will graph say, 1 hour on either side of every note’s time frame, making it easy to instantly populate graphs with important data without having to search every time. I’ll be giving it more thought for sure. 

Since I seem to like ordering my options in the config and sensor files, in a way that’s not great for compatibility (AKA I don’t add every new option to the bottom, I reorder the whole list).  This is a pain, as I manually have to log in through SSH and re-adjust the sensor configuration files for the new options.  So I had a few ideas on solving it. 
1) Add the ability to change installed sensors from the control center
and or
2) Create some updated functionality to automatically make needed changes after the upgrade. 
I went with the 2nd option first and made an upgrade module that adjusts any needed files and related settings, then re-saves them.  It also uses the versioning system, so it can incrementally update from this version onwards, even if you miss a few versions along the way!  This will make my future upgrade tests go a bit smoother!

My other major upgrade was to the environmental temperature adjustments.  Before it was a manually set option in the graph window, but this makes it annoying to swap between different sensors with different offset needs.  So I moved the offset into the sensor itself and created different offsets for different setup combinations.  This way it’s all automatically set by how you set your installed sensors configuration file!  AKA it will give recommended offsets depending on what hardware is being used.  So now you can check a “Use Default” temperature offset checkbox, and the graphs will use the offset provided by the sensor.  A custom offset is also available in the sensor configuration file, in case a sensor doesn’t follow standards.  And of course you can uncheck the use default and set your own setting during graphs.  Whatever is set on the sensor, will also be saved into the SQL database when the environmental temperature readings are saved.  

Those are the 3 main things I have been working on.  

I think I’ll re-work my re-configuration of the sensors.  I think I can make a decent live update by pulling and pushing the text files into and out of a textbox, similar to my Log viewer and Notes window.  If I do that, I think the program can do everything remotely!  Between the log viewer, update module and soon to be installed sensors and config adjuster, keeping my demo and test sensors running smoothly through upgrades is starting to sound like an easier task 🙂

Nicer touches

I have re-visited my Sensor Readings Report code and changed it to create all the readings in a table for lining up.  I also put each sensor into its own bordered box to help separate sensors.  While I was doing that, I figured out how to get multiple packets of data with sockets.  Before, all my data was under 4096 bytes, except for my logs, which I only needed the end of anyway.  However, I can now get a bit more of the logs and ensure data is properly received. I had some trunicated errors doing Reading reports randomly before and it caused the socket to die, requiring a timeout.  Now it functions as intended and no delay in restarting occurs.  I also added the ability for the socket to be rebonded right away if the application is terminated unexpectedly.  This should speed up Sensor Command restarts after remote upgrades.   

I have now set up a Pi + Goal Zero Sherpa 50 with a solar panel keeping it charged in the window.  This should give me a good idea if it will work as a self-sustaining power system for off the grid sensors.  I’m told windows can block a lot of sun rays, making solar charging less efficient, so if it works there, it will work even better outside.  I still have to figure out how to keep the snow off the solar, but I do have an idea… angled setups and mirrors would probably allow it.  Multiple mirrors set up in different spots could also help with “temporary intensive charging”, to help give it a boost at certain times, which could help during long dark periods… I’m just not sure how long they will be effective before I have to adjust the positions of the mirrors, due to sun path changes through the seasons.  

Feeling pretty good about the progress again.  It was nice to take a bit of a break, but I would like to try and do at least a bit of programing every 2nd day to maintain forward momentum.  

Hope everyone had a great weekend!

SenseHAT Ehancements + Updates

If you are lucky enough to have a SenseHAT with your Raspberry Pi, I have updated the sensor software to allow the use of its JoyStick and LED grid! The following commands are

  • Up = Scroll IP address on the LED grid
  • Down = Display “Steve” on the LED grid OR Confirms Shutdown
  • Left = Scroll humidity
  • Right = Scroll Temperature
  • Push = Initiate Shutdown (Requires confirmation of Down JoyStick)

Due to the extra additions on the SenseHAT, it is a nice sensor to start with for trying things out, since you can just plug in power and be able to see readings right on the LED grid AND have it shut down with the button combo.  Other sensors you have to shutdown remotely with the Control Center or login directly and shut down, and of course, no other currently supported sensor has a display. 

I now have a working Gnu/Linux installer working for both SMB and HTTP, but customizing a file is required if you want the desktop shortcuts on a different user than pi (Menu shortcuts work just fine). 

In other news, a few new battery backup systems have come in for me to try out.  This time I decided to go with something a bit more expensive to help with the solar deployments.  To that end, I grabbed a “Goal Zero Yeti 150” and a “Goal Zero Sherpa 50”.  They both have hookups for solar panels and are pretty hefty.  I put a RP Zero + Garden HAT on the bigger 150 to see how long it lasts.  I’m going to put the smaller one outside with a solar panel and see how that goes.  Both units oddly enough cost me about $250 each.  If they work as fully self-sustaining units, they are worth the price.  O yeah, the manual for the Yeti was pretty useful! It says to help with using the battery in colder temperatures, put it in a cooler, as the ambient temperature it puts off would keep it warm enough inside the insulated container.  A very good idea, since it will be in the outdoors during winter and I have been thinking of ways to protect the battery units from the elements.  I look forward to testing results, but I suspect I’m going to have trouble keeping the snow off the solar panels … I might have to be a bit more creative for that.  

That’s it for tonight.  

Version 21.18 Released!

New Version out!  I was going to get pipenv working before release… but it looks like that’s going to take a bit more time.  
Here are a few highlights of this release.

  • Allow Live Graph on 3 point data
  • Merged Sensor SQL Interval & Trigger code & databases
  • Created Log viewer and downloader in Control Center
  • Tweaked System Report and added Installed Sensors to Config Report
  • Raspbian desktop and menu shortcuts added
  • Created uninstaller for sensors
  • Changed locations for sensor programs, configuration and data to be more in line with Gnu/Linux file structuring
  • Updated help files
  • Bug fixes and code optimization

Due to the location and database structure changes, this version is not compatible with older versions.  I recommend doing a clean install if you have  a previous version installed. 


I spent some more time refactoring the Raspberry Pi Sensor code.  For one, I have the import code for each sensor, in the init part of the sensor access class, that’s created per sensor.  AKA, creating a class instance of a sensor, will then pull in the needed libraries at that time.  I did this originally so unneeded sensor libraries would never be imported, saving time and resources.  However, I then realized that I create the sensor object every time any sensor data is taken, which re-initializes the sensor every time!  Long story short, I moved the sensor access object instance creations out of the sensor retrieval functions and into the primary python program before the main loop.  It only initializes sensor access for sensors specified in the “Installed Sensors” configuration file.  This means sensor readings taken, will re-use the same sensor access instance that was created at the start of the program and only create access instances for installed sensors. 

Doc strings have been added to all functions and classes in the raspberry pi sensor code, and I started to get pipenv working for the package management and virtual environments for my programs… but it keeps erroring out on pipenv install.  It almost seems like its timing out on installing the packages, but I can’t be sure since I’m not sure where it stores the logs if it creates any.  In theory, it sounds like it will simplify the install process and help keep a consistent base to run on, but it looks like it will take a bit more time to have it function as intended.  

I was hoping to get pipenv working before creating new Gnu/Linux install scripts, and I was hoping to have the install scripts working before my next release.  So I’ll continue to diagnose the pipenv issues, in hopes that I can use it for with the new installers.  

Hope everyone’s weekend went well!

More SQL write modifications

I figured it was about time to re-look at the “write to SQL DB” code, especially for the datetime entries when writing to the database.  Long story short, I had SQL input the datetime values at the time of recording, but because the code grabbed a few readings to check against the variances, the data ended up being entering with fairly inaccurate datetime correlations (by hundreds of milliseconds or more).  

So, I changed the code in a few ways.  First, I found out how to get the exact same formatting for the datetime as SQL was using, and used that at the time of actually getting the sensor data (through Python datetime).  I also made the Trigger variance checks & database recording, work on 5 readings at a time.  If any of the 5 readings trigger a recording, all 5 will be recorded, so in theory, it will record entries before and after the event that triggered it. 

That’s it for today. 

Live Graph updated

w00t, the Live Graph now works with all supported sensors, AKA Live Graphs work on sensors that give 3 data points as well as just one. I also changed the layout to make it look “cleaner”.  

I have been thinking of a few other features to add, like LED / LCD screen support for displaying sensor readings on the sensors themselves, making the joystick on the SenseHAT functional and adding default reading offsets to the sensors themselves, and sending it with the sensor data for whatever reading it might be.  This will allow the Control Center to choose between the hardware default and a custom setting.

In other news, I have put a few more sensors around my house, to see how the temperature fluctuates in different parts of the house, during different times of the day and night.  The recordings should show me how the heat moves to different parts of the house, at what temperature and how long it takes to get up to temperature, and if certain rooms get more or less heat during the process.  With winter coming and living in a Mobile Home, there should be some good differences!

I have been testing some power systems for the Raspberry Pi, so that I can ultimately have a solar powered system, that charges a battery to keep it running at night.  I tried the “MP2636 Power Booster”, but had mixed results… it seems OK on my one Raspberry Pi 3B+, but it failed to stay on or even charge the battery for a few units…. I suspect polarity is wrong on my cheaper china batteries, but that doesn’t explain the other RP Zero W rebooting when swapping the primary power source.  Long story short, I don’t think the “MP2636” is going to work for my needs.  Next up is one made specifically for the Pi Zero, called “Juicebox Zero”.  It looks very promising, but I have to solder it on, so I’m putting it off for a bit.

** Few days later ** 

Guess I forgot to actually post this a few days back.  I have made some other major changes since the top half of this post.  For starters, I just did an overhaul of the write to SQL Database code.  I merged my 2 programs into one, so now both Trigger and Interval code run from the same python program.  I was mildly torn about doing this, but I think it’s ultimately for the best, not only on system resources and response times but for dealing with the readings as well, as you can now graph both Interval and Trigger readings from the database to a single graph page.  It’s also much nicer to deal with a single database for downloading and storing.

I have changed the “Clean” update code to run as a service (Manual start one-time run), this allows me to fully delete the main folders and files, without terminating the very program that’s doing it, which was preventing the re-install process after deleting everything… AKA it would delete everything and stop.  Now, the program initiates the update service to start, which keeps running after the program that started it, is terminated.  

What else was done… O yes, I added what sensors are installed in the Configuration report, worked on the Unit Testing and did a fair amount of refactoring.  I’m sure there were other things too, but I can’t think of them right now.  

Since I did a few overhauls in the code, I’m going to test things a bit more before an updated release.