Monthly Archives: December 2018

Potly Graph updates

So I thought it was about time to update the offline Plotly graphing module to allow the new sensors. While I was in there, I thought I might as well update a few other things. So I added some consistent markers to the XYZ readings, replaced CPU rendered graph with OpenGL (WebGL) rendering for Windows (doesn’t work properly on the Raspberry Pi).

I have to say, the OpenGL rendering is SOO much faster than using CPU rendering. I can now easily zoom in and out of really large data sets. Large for me being over 115,000 data points per sensor entry. When I do the same graph with CPU graphing, it takes more than 2 min to render at all, and I never got to the point I could zoom in. Running my mouse over readings to see the values and datetime stamp lagged a lot too. So yay for GL rendering! I just hope its fully supported on the Raspberry Pi later on, as it makes a big difference!

In other news, it was nice and sunny today! I spent a decent amount of time outside with my daughter, absorbing as much sun as I could while she played on the big snow piles.

It may take a bit longer to finish the new trigger recording code should I decide to re-work how its done … which I might … I don’t plan on changing what it does, just how it does it… I’m pretty sure I could use a few lists or a class with a ‘for’ statement or two in order to shorten the length of the code.

That’s it for now. I’ll probably release another version once I fully finish up the new trigger sensor recording. The winter continues to slow progress, but I’m still poking the code every now and then!

Finally done the trigger overhaul

OK, it did take a bit longer to get the trigger recording overhaul done, but mostly due to the downturn of motivation, however, it is all done now! well not ‘done done’, as I still have to add additional sensors to the trigger checks and make configuration options for them, but Interval recording is now recording all sensor readings and Trigger recording is working like before, but ready to have additional sensors added. I also took the time to do more code refactoring, such as creating folders for holding the program modules and consolidating more options into the configuration py file.

Off to do more holiday stuff with the family.

Re-Thinking Interval & Triggers

After a bit of thought, I think it makes sense to have ALL sensors available as both Interval and triggers. So that’s my next overhaul. I’m also going to add the option in config to enable to disable each one individually as well. For simplicity and making sure the sensor doesn’t start filling the storage too quickly, I’ll have Intervals enabled by default @ 5 min and Triggers disabled by default. Since I already have the 3 data point checks in place, it will be extra easy to add single checks … in fact, I’m pretty sure I can just make a single ‘single check’ 🙂 Pass the type and sensitivities as variables and bobs your uncle!

On another note, I have done preliminary work on the SQL modifier. I drew up a design idea about how to display and interact with the data. So I’ll start up the GUI soon… I’m just debating integrating it into the main control center program, instead of having it separate. I’ll be sure to make it modular as I go, so it can be integrated easily later if I so choose.

1 day later …

So I’m halfway through my Interval and Trigger modifications. There’s actually a bunch of other refactorings that have to be done to make it all work. Nothing too complicated, but I do have to step back a bit to see the big picture and figure out the best way to have it all mesh and work together.

I also took the time to re-organize my file structures for the rp-sensor program. I moved all the operation py files into a operations folder. It’s a lot cleaner now since many operational files have been added over time.

The development version of my program is broken until I get the modifications done, so I’ll most likely have it done by tomorrow (I don’t like leaving things broken). Who knows, maybe I’ll even finish it tonight … but I doubt it, since my brain is not working very well today.

In other news, I have set up my Goal Zero Yeti 150 in the shed plugged into power to keep charged. So it should keep my sensors in the back up during power outages. Later, when there is more sun, I’ll see about setting up the solar panels to keep it charged instead. It has been really dark recently, not only due to it being the shortest day of the year soon but also it has been darn cloudy. I miss summer and fall already… well maybe not the superheat of summer, but definitely the bright sun during it.

Slow winter progress but progressing all the same. At some point down the road, I also want to start my IT program, which I will use as my quick goto utility to get system information and run common commands. I might even make it with Flask or Django, so it has the ability to be used remotely later on.

I hope everyone has a great winter holiday!

New Mini Router ‘n’ Stuff

As the title suggests, I have a new mini router for my sensors.  Same company, but a version with 2 external antennae.  It works MUCH nicer!  My far sensors in the shed connect with no issues now!  Before, pings were sporadically 10ms up to 300ms, but with this new mini router, it’s pretty much a consistent 5ms for all the sensors.  Lost pings also went from 33% in the shed to about 1%.  
It costs about double the price ($55 Canadian vs. about $25 for the original) but it is worth it in my case. 

I randomly had some Refactoring inspiration and motivation, so I jumped on that yesterday.  I also ‘fixed’ a few spots with graphing, such as not trying to graph at all if they cancel the database selection.  I also put in a few more pop-up messages for the user when things are cancelled or not selected right.

I have been thinking of what would make a release worthy of a Beta or pre-release, and without actually researching standards, I think when I’m certain about the ‘core’ operations of the program, I’ll shift it to Beta.  While I’m still deciding or open to changes on things like my network protocols, database column names, how data is stored, etc, etc, it will remain Alpha.  

In other news, I’m not super enjoying winter right now, hovering just around 0 Celcius.  Seems just right for creating really wet and slushy conditions outside.  It makes it hard for me to enjoy my outdoor walks and jogs.  So I’m guessing program progress will suffer a bit (not to mention my mood).  However, the wonderful thing about making programs, is once you get aspects right once, its right from then on (unless changes want to be made).  So even if I only do a little, over time it will get better and better.  

Alpha.22.24 Released

New version out!  According to my website, it’s been exactly 1 month since the last release.  There have been a lot of changes since the last one, which includes a change from python sockets to Flask.  So this version is very much not compatible with the previous, as far as the Control Center communication goes.  The bigger changes include the following.

  • Change from Sockets to Flask
    • Starting WebApp to control sensors with a web browser
  • Enabled support for 2 new Pimoroni sensors
  • Sensor Configuration now allows Config & Installed sensors to be adjusted
  • Added 6 channel support to the colour spectrum recording and Live graphing
  • Added sensor provided temperature offset, determined by installed hardware.  This is also recorded to the database
  • Refactoring, bug fixes & tweaks

One thing that is missing is the ability to do a Plotly graph with all 6 colour channels, but it still graphs the RGB of a 6 channel sensor just fine. I have also changed the python virtual env, included in the windows installers, so you can’t do an upgrade from previous versions.  The RP sensors should upgrade just fine though. 

That’s it for now.  

Updates!

I added support for my new sensors, and got the live graph to show all 6 colours of the new 6 channel spectral sensor.  

All good news, but I feel like I’m starting to get a bit lazy on my implementations, and I need to set aside some time to refactor.  I have already started a bit here and there.  For example, I have been moving stray configuration options and settings into the config python file.  I want to make it the one-stop file for almost all the customized variables.  I also need to get my unit tests working again, since the move to Flask.  

There has been a lot of additions since the last release now, so I’m trying to work on tests instead of features … but I do kinda want to get the SQL note viewer and editor working.  I’m pretty sure no one actually uses my program yet, so I might hold off on the release and work on a few more features and some refinement.  

In other detailed adjustments, I was thinking of how to add the different Wavelengths of light into the database.  Since I have RGB saving into the database as individual colours “Red, Green, Blue”, I finally decided on adding the 6 common colours of the visible spectrum and make notes in the sensor module for the Wavelengths it’s capturing.  This should allow a balance of accuracy and simplicity.  I was thinking of using the Wavelength itself as the entry column but realized it would add extra complications down the road.  

I have definitely slowed down on progress, and I think the winter blues are causing a down turn of motivation.  

New Sensors and work

So, good news first, I just got some new sensors as of … 30 min ago?  I’m really excited to test out the 6 channel spectrometer, but I also got a light and proximity sensor, Garden HAT for the Pi Zero and a few LCD displays.
Specifically, I got the following from Pimoroni.

  • AS7262 – 6 channel spectral sensor
  • LTR-559 – Close range (4 cm?) proximity detector & lumen sensor
  • 1.12″ Mono OLED 128×128 display

In other news, I revisited my Raspberry Pi Network Testers, which are 2x Pi’s made to test Ethernet cables and routes.  It’s now been updated to my present Python & Linux programming knowledge.  AKA it now uses systemd services, bash installer scripts and works directly with the original E-Ink libraries, instead of a 3rd party one I found.  All in all, its cleaner, more efficient and functions as one would expect for use and errors.  

I’m getting closer to releasing another Sensor Control Center update, but I might tweak a few things before that.  I want to enhance my SQL note section, so you can browse, edit and delete entries.  
We’ll see what happens. 

I have also been thinking about making an I.T. technician toolkit of sorts.  Creating a simple GUI app to quickly do common IT related tasks, like quick virus scan with a self-contained scanner, ccleaner type clean-ups, show system information like IP, uptime, free mem and HDD, etc.  I think I’ll also create a built-in script runner, so you can pick from a dropdown to run whatever script you like, as well as the ability to add new ones.  If I create most things as scripts, I should be able to do almost anything … So I’m kinda looking forward to releasing a useful diagnostic tool.  

Since I have been a bit busy with work, I have not been working as much on my programs.  

Until next time!

Minor changes & stuff

I have been working to update the smaller stuff after those overhauls before.  Editing comments, updating help files and fixing bugs.  

Now that I have Flask in use, it allows me to show reports off the sensor itself, by going to a URL with the sensors IP (or hostname if DNS is setup right).  I’m working on http://sensor_ip:port/TestSensor.  I created a combo of all 3 Control Center reports in one.  Going to that URL with an actual IP and the right port would give a nice combo HTML report of the sensor, complete with name, version, current sensor readings, configuration setups, etc!  
The big thing I like about this is you can take your cell phone, make it a wifi hotspot the sensors connect to and goto the URL in a web browser on said phone to see its status.  If I make an actual phone app, I may be able to automate the process with a push of a button.  Cool possibilities for the future! 

I have been pondering some other back-end changes.  Since I know a lot more about python now, I might move a bunch of my install code from the bash script into the program’s first launch.  I should probably keep the bash script to a bare min, just enough to get the program needs installed, then let the program start and take over from there.  

I should probably set a time to do a purely refactoring session too.  There are a lot of places where the code could use improvement where I was trying to get it working at all.  I also really want to get the Live Graph working with threading for the data gathering.  Since right now, if the connection is not so great, the program will freeze until it times out.  So what I’m thinking of doing, is simply threading the data gathering and putting results in a queue or pool of sorts.  So long as this pool allows the program to continue while waiting for data, it should work very well.  Otherwise, I’ll set aside some time to really think about how to make it shine … or maybe replace it… Since Matplotlib is not thread safe to begin with, maybe I should look for something that is more like a multi-camera dashboard.  I’m pretty sure most people are going to want to put a dashboard of live sensors up in front of them, to monitor things like crops and such.  

I will most likely release a new version soon, as there are a lot of nice touches on this one, such as most areas are threaded to prevent program freezes now (minus the live graph), the program is no longer dependant on a 3rd party HTTP server, the Control Center sensor re-configuration section has been re-done to allow config and installed sensor changes. It also allows grabbing a sensor configuration to compare or re-distribute to other sensors.  Graphing now allows a default temperature offset from the sensor itself, which has an automatic offset based on the hardware chosen, as well as allows a custom entry.  Notes can now be entered into the Sensor Database with a date and time stamp so it will (eventually) show up in graphs beside relevant sensor data. I have created an update and program checks that will make needed adjustments to settings and files after an upgrade.  Hmm, what else… O yeah, I was able to remove the rebuild of numpy with the new Raspbian version that’s out, but I had to add ‘atlas’ i think… but just for install, not rebuild it, so that’s good.  

Lots of good improvements and it seems to be running fairly smoothly with the transition from sockets to Flask… although I must admit, I do notice a slight more delay using Flask, which makes sense due to the overhead.  The delay is also measured in ms not seconds, so its not bad. It is especially hard to notice now that I have added threading to pretty much every sensor reading grab, so it shows up as a delay in showing, not a frozen program.   

That’s it for now.