Monthly Archives: November 2018

Flask seems a good fit

I was looking for a better way to do my Network Server commands, so I headed over to the Python IRC channel and asked for some better ways to send text across the network.  After some chatting back and forth someone mentioned HTTP and REST, so I started my research.  It didn’t take long to find Django (what I used before) and Flask (I saw before, but haven’t used).  So I looked around trying to see what the purpose and differences of the 2 are.  It was obvious pretty quickly that Flask was the one.  My commands and data are pretty basic, so having a micro framework to send and receive made sense.  There also seems to be a lot of module attachments to enhance Flask, so future expansion is there.  

I have now replaced my basic sockets server with Flask.  I have to admit, there’s a lot to like.  For one, I can just go to the URL of the sensor in a web browser to view results!  Soo very helpful during testing to see if anything looks off.  This also opens up possible phone apps that just use a wrapper around a web browser.  It would be pretty sweet to use my phone to interact with sensors.  

Definitely a few more possibilities in the near future.  I still have to rework some of my unit tests and actually test the program a bit more, since I just overhauled the network service, but so far it’s looking pretty good.  

Lots to do, so back to it I go. 

Update system & notes!

I have been working on a few things as of late.  For one, notes can be saved into the sensor Database at whatever time you choose.  This is meant to aid in putting notes beside the actual sensor data.  I’ll be adding the notes into the graphs later, so that they will show up with the graphed sensor data, using the time you enter!  AKA, you notice the sensor malfunctioning at a certain time, you enter a note with the same or close Date & Time saying “Malfunction, ignore this” or “Very important data represents time travel”.  Whatever helps identify what the data represents.  It can also be used to bookmark or tag certain times with things, like manual settings for temperature offset or other important information not being captured by the sensors, etc.  Later on, I’ll try to get it so you can “Graph by notes”, and it will graph say, 1 hour on either side of every note’s time frame, making it easy to instantly populate graphs with important data without having to search every time. I’ll be giving it more thought for sure. 

Since I seem to like ordering my options in the config and sensor files, in a way that’s not great for compatibility (AKA I don’t add every new option to the bottom, I reorder the whole list).  This is a pain, as I manually have to log in through SSH and re-adjust the sensor configuration files for the new options.  So I had a few ideas on solving it. 
1) Add the ability to change installed sensors from the control center
and or
2) Create some updated functionality to automatically make needed changes after the upgrade. 
I went with the 2nd option first and made an upgrade module that adjusts any needed files and related settings, then re-saves them.  It also uses the versioning system, so it can incrementally update from this version onwards, even if you miss a few versions along the way!  This will make my future upgrade tests go a bit smoother!

My other major upgrade was to the environmental temperature adjustments.  Before it was a manually set option in the graph window, but this makes it annoying to swap between different sensors with different offset needs.  So I moved the offset into the sensor itself and created different offsets for different setup combinations.  This way it’s all automatically set by how you set your installed sensors configuration file!  AKA it will give recommended offsets depending on what hardware is being used.  So now you can check a “Use Default” temperature offset checkbox, and the graphs will use the offset provided by the sensor.  A custom offset is also available in the sensor configuration file, in case a sensor doesn’t follow standards.  And of course you can uncheck the use default and set your own setting during graphs.  Whatever is set on the sensor, will also be saved into the SQL database when the environmental temperature readings are saved.  

Those are the 3 main things I have been working on.  

I think I’ll re-work my re-configuration of the sensors.  I think I can make a decent live update by pulling and pushing the text files into and out of a textbox, similar to my Log viewer and Notes window.  If I do that, I think the program can do everything remotely!  Between the log viewer, update module and soon to be installed sensors and config adjuster, keeping my demo and test sensors running smoothly through upgrades is starting to sound like an easier task 🙂

Nicer touches

I have re-visited my Sensor Readings Report code and changed it to create all the readings in a table for lining up.  I also put each sensor into its own bordered box to help separate sensors.  While I was doing that, I figured out how to get multiple packets of data with sockets.  Before, all my data was under 4096 bytes, except for my logs, which I only needed the end of anyway.  However, I can now get a bit more of the logs and ensure data is properly received. I had some trunicated errors doing Reading reports randomly before and it caused the socket to die, requiring a timeout.  Now it functions as intended and no delay in restarting occurs.  I also added the ability for the socket to be rebonded right away if the application is terminated unexpectedly.  This should speed up Sensor Command restarts after remote upgrades.   

I have now set up a Pi + Goal Zero Sherpa 50 with a solar panel keeping it charged in the window.  This should give me a good idea if it will work as a self-sustaining power system for off the grid sensors.  I’m told windows can block a lot of sun rays, making solar charging less efficient, so if it works there, it will work even better outside.  I still have to figure out how to keep the snow off the solar, but I do have an idea… angled setups and mirrors would probably allow it.  Multiple mirrors set up in different spots could also help with “temporary intensive charging”, to help give it a boost at certain times, which could help during long dark periods… I’m just not sure how long they will be effective before I have to adjust the positions of the mirrors, due to sun path changes through the seasons.  

Feeling pretty good about the progress again.  It was nice to take a bit of a break, but I would like to try and do at least a bit of programing every 2nd day to maintain forward momentum.  

Hope everyone had a great weekend!

SenseHAT Ehancements + Updates

If you are lucky enough to have a SenseHAT with your Raspberry Pi, I have updated the sensor software to allow the use of its JoyStick and LED grid! The following commands are

  • Up = Scroll IP address on the LED grid
  • Down = Display “Steve” on the LED grid OR Confirms Shutdown
  • Left = Scroll humidity
  • Right = Scroll Temperature
  • Push = Initiate Shutdown (Requires confirmation of Down JoyStick)

Due to the extra additions on the SenseHAT, it is a nice sensor to start with for trying things out, since you can just plug in power and be able to see readings right on the LED grid AND have it shut down with the button combo.  Other sensors you have to shutdown remotely with the Control Center or login directly and shut down, and of course, no other currently supported sensor has a display. 

I now have a working Gnu/Linux installer working for both SMB and HTTP, but customizing a file is required if you want the desktop shortcuts on a different user than pi (Menu shortcuts work just fine). 

In other news, a few new battery backup systems have come in for me to try out.  This time I decided to go with something a bit more expensive to help with the solar deployments.  To that end, I grabbed a “Goal Zero Yeti 150” and a “Goal Zero Sherpa 50”.  They both have hookups for solar panels and are pretty hefty.  I put a RP Zero + Garden HAT on the bigger 150 to see how long it lasts.  I’m going to put the smaller one outside with a solar panel and see how that goes.  Both units oddly enough cost me about $250 each.  If they work as fully self-sustaining units, they are worth the price.  O yeah, the manual for the Yeti was pretty useful! It says to help with using the battery in colder temperatures, put it in a cooler, as the ambient temperature it puts off would keep it warm enough inside the insulated container.  A very good idea, since it will be in the outdoors during winter and I have been thinking of ways to protect the battery units from the elements.  I look forward to testing results, but I suspect I’m going to have trouble keeping the snow off the solar panels … I might have to be a bit more creative for that.  

That’s it for tonight.  

Version 21.18 Released!

New Version out!  I was going to get pipenv working before release… but it looks like that’s going to take a bit more time.  
Here are a few highlights of this release.

  • Allow Live Graph on 3 point data
  • Merged Sensor SQL Interval & Trigger code & databases
  • Created Log viewer and downloader in Control Center
  • Tweaked System Report and added Installed Sensors to Config Report
  • Raspbian desktop and menu shortcuts added
  • Created uninstaller for sensors
  • Changed locations for sensor programs, configuration and data to be more in line with Gnu/Linux file structuring
  • Updated help files
  • Bug fixes and code optimization

Due to the location and database structure changes, this version is not compatible with older versions.  I recommend doing a clean install if you have  a previous version installed. 

Stuff

I spent some more time refactoring the Raspberry Pi Sensor code.  For one, I have the import code for each sensor, in the init part of the sensor access class, that’s created per sensor.  AKA, creating a class instance of a sensor, will then pull in the needed libraries at that time.  I did this originally so unneeded sensor libraries would never be imported, saving time and resources.  However, I then realized that I create the sensor object every time any sensor data is taken, which re-initializes the sensor every time!  Long story short, I moved the sensor access object instance creations out of the sensor retrieval functions and into the primary python program before the main loop.  It only initializes sensor access for sensors specified in the “Installed Sensors” configuration file.  This means sensor readings taken, will re-use the same sensor access instance that was created at the start of the program and only create access instances for installed sensors. 

Doc strings have been added to all functions and classes in the raspberry pi sensor code, and I started to get pipenv working for the package management and virtual environments for my programs… but it keeps erroring out on pipenv install.  It almost seems like its timing out on installing the packages, but I can’t be sure since I’m not sure where it stores the logs if it creates any.  In theory, it sounds like it will simplify the install process and help keep a consistent base to run on, but it looks like it will take a bit more time to have it function as intended.  

I was hoping to get pipenv working before creating new Gnu/Linux install scripts, and I was hoping to have the install scripts working before my next release.  So I’ll continue to diagnose the pipenv issues, in hopes that I can use it for with the new installers.  

Hope everyone’s weekend went well!

More SQL write modifications

I figured it was about time to re-look at the “write to SQL DB” code, especially for the datetime entries when writing to the database.  Long story short, I had SQL input the datetime values at the time of recording, but because the code grabbed a few readings to check against the variances, the data ended up being entering with fairly inaccurate datetime correlations (by hundreds of milliseconds or more).  

So, I changed the code in a few ways.  First, I found out how to get the exact same formatting for the datetime as SQL was using, and used that at the time of actually getting the sensor data (through Python datetime).  I also made the Trigger variance checks & database recording, work on 5 readings at a time.  If any of the 5 readings trigger a recording, all 5 will be recorded, so in theory, it will record entries before and after the event that triggered it. 

That’s it for today. 

Live Graph updated

w00t, the Live Graph now works with all supported sensors, AKA Live Graphs work on sensors that give 3 data points as well as just one. I also changed the layout to make it look “cleaner”.  

I have been thinking of a few other features to add, like LED / LCD screen support for displaying sensor readings on the sensors themselves, making the joystick on the SenseHAT functional and adding default reading offsets to the sensors themselves, and sending it with the sensor data for whatever reading it might be.  This will allow the Control Center to choose between the hardware default and a custom setting.

In other news, I have put a few more sensors around my house, to see how the temperature fluctuates in different parts of the house, during different times of the day and night.  The recordings should show me how the heat moves to different parts of the house, at what temperature and how long it takes to get up to temperature, and if certain rooms get more or less heat during the process.  With winter coming and living in a Mobile Home, there should be some good differences!

I have been testing some power systems for the Raspberry Pi, so that I can ultimately have a solar powered system, that charges a battery to keep it running at night.  I tried the “MP2636 Power Booster”, but had mixed results… it seems OK on my one Raspberry Pi 3B+, but it failed to stay on or even charge the battery for a few units…. I suspect polarity is wrong on my cheaper china batteries, but that doesn’t explain the other RP Zero W rebooting when swapping the primary power source.  Long story short, I don’t think the “MP2636” is going to work for my needs.  Next up is one made specifically for the Pi Zero, called “Juicebox Zero”.  It looks very promising, but I have to solder it on, so I’m putting it off for a bit.

** Few days later ** 

Guess I forgot to actually post this a few days back.  I have made some other major changes since the top half of this post.  For starters, I just did an overhaul of the write to SQL Database code.  I merged my 2 programs into one, so now both Trigger and Interval code run from the same python program.  I was mildly torn about doing this, but I think it’s ultimately for the best, not only on system resources and response times but for dealing with the readings as well, as you can now graph both Interval and Trigger readings from the database to a single graph page.  It’s also much nicer to deal with a single database for downloading and storing.

I have changed the “Clean” update code to run as a service (Manual start one-time run), this allows me to fully delete the main folders and files, without terminating the very program that’s doing it, which was preventing the re-install process after deleting everything… AKA it would delete everything and stop.  Now, the program initiates the update service to start, which keeps running after the program that started it, is terminated.  

What else was done… O yes, I added what sensors are installed in the Configuration report, worked on the Unit Testing and did a fair amount of refactoring.  I’m sure there were other things too, but I can’t think of them right now.  

Since I did a few overhauls in the code, I’m going to test things a bit more before an updated release.